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EXECUTIVE  SUMMARY 


UAVs  (Unmanned/uninhabited  aerial  vehicles)  are  an  increasingly  important  part  of  military 
operations  throughout  the  world.  However  there  is  no  consensus  about  who  should  fly  these  aircraft. 

The  United  States  Air  Force  (USAF)  uses  experienced  pilots  who  have  already  had  at  least  one 
operational  tour  of  duty  in  another  combat  aircraft.  By  contrast,  the  U.  S.  Army’s  Hunter  UAV  is  flown 
by  enlisted  personnel  (generally  non-pilots)  who  are  given  a  UAV-specific  training  program.  Other 
possibilities  include  students  at  various  stages  of  military  pilot  training,  or  civilian  pilots  under  contract 
to  the  military.  A  USAF  Corona  South  four-star  general  officer  summit  in  1997  resulted  in  tasking  the 
Air  Force  Research  Laboratory  (AFRL)  to  conduct  a  study  to  compare  the  speed  and  accuracy  with 
which  various  groups  of  pilots  could  learn  to  fly  the  RQ-1A  Predator  UAV.  This  report  contains  the 
procedures  and  results  of  that  study. 

For  this  study  we  used  a  high-fidelity  RQ-1 A  simulator  developed  in  cooperation  with  experi¬ 
enced  Predator  pilots  at  Indian  Springs  Air  Force  Auxiliary  Field  (AFAF).  Seven  groups  of  pilots, 
varying  in  amount  and  kind  of  flying  experience,  completed  a  series  of  multimedia  tutorials  on  basic 
principles  of  flight  and  procedures  for  operating  the  Predator,  then  flew  the  simulator.  These  groups 
were:  (a)  experienced  USAF  Predator  pilots;  (b)  experienced  USAF  pilots  recently  selected  to  fly 
the  Predator;  (c)  students  recently  completing  USAF  T-38  training;  (d)  students  recently  completing 
USAF  T-l  training;  (e)  students  recently  completing  single-engine  instrument  training  at  Embry-Riddle 
Aeronautical  University  (ERAU);  (f)  students  recently  completing  requirements  for  a  private  pilot’s 
license;  and  (g)  Reserve  Officer  Training  Candidate  (ROTC)  students  at  ERAU  who  intend  to  be  USAF 
pilots  but  who  had  no  flying  training  or  experience.  Each  participant  flew  basic  maneuvers  and  landings 
(including  difficult  crosswind  landings)  until  a  very  high  standard  of  aircraft  control  performance  was 
achieved,  then  flew  30  reconnaissance  scenarios.  All  of  this  required  between  12  and  30  hours,  depend¬ 
ing  predominately  on  the  skill  of  the  participant.  During  this  time,  detailed  measures  of  performance 
were  continuously  and  automatically  recorded. 

The  results  show  that  experienced  Predator  operators  performed  consistently  better  than  other 
groups,  while  the  nonpilot  ROTC  students  performed  worse.  This  is  not  surprising,  but  it  confirms 
that  the  simulation  used  in  the  study  was  a  valid  indicator  of  Predator  stick-and-rudder  skill.  A  more 
surprising  finding  is  that,  from  a  pure  aircraft  control  perspective,  T-38  graduates  and  civilian  instrument 
pilots  performed  nearly  as  well  as  the  much  more  experienced  pilots  currently  selected  for  Predator 
training.  A  possible  explanation  for  relatively  good  performance  of  the  T-38  and  civilian  instrument 
pilots  is  that  there  may  be  some  advantages  to  recent  experience  flying  aircraft  that  have  handling 
characteristics  that  are  similar  to  the  Predator  in  some  ways. 

This  study  applies  only  to  the  Predator,  and  primarily  addresses  stick-and-rudder  skills;  we 
did  not  measure  such  operationally  relevant  factors  as  communication  skills,  command  experience, 
knowledge  of  combat  operations,  or  familiarity  with  airspace  management,  especially  in  war  zones. 
Since  such  knowledge  and  skills  are  necessary  for  a  Predator  pilot,  the  current  USAF  policy  of  using 
pilots  who  have  had  prior  operational  tours  should  only  be  altered  if  it  can  be  assured  that  less 
experienced  pilots  can  achieve  this  critical  competence  in  some  other  way. 
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INTRODUCTION 


The  role  of  unmanned/uninhabited  aerial  vehicles  (UAVs)  is  expanding  rapidly  in  military  opera¬ 
tions  after  a  period  of  dormancy.  UAVs  are  becoming  a  mainstay  of  intelligence,  surveillance,  and 
reconnaissance  (ISR)  information  gathering,  with  the  capability  of  supplying,  in  near  real-time,  visual 
information  to  theater  command  and  control  centers  and  to  pilots  in  their  cockpits.  The  number  and 
type  of  UAVs  is  also  expanding.  UAVs  may  become  increasingly  involved  in  more  direct  combat 
roles  involving  weapons  deliveries  for  various  air  combat  missions.  Already,  the  United  States  Air 
Force  (USAF)  Predator  (officially  designated  RQ-1)  has  been  used  to  deliver  Hellfire  missiles  and  to 
laser-designate  targets.  The  USAF  has  the  Unmanned  Combat  Air  Vehicle  (UCAV)  in  development. 

The  UCAV  has  had  a  weapons  delivery  capability  from  its  original  conception.  The  Global  Hawk,  a 
high-altitude  ISR  aircraft,  is  nearing  an  initial  operational  capability  (IOC)  date.  The  other  services  have 
a  variety  of  other  UAVs  in  service  and  many  are  on  the  drawing  board  for  the  future.  In  1996,  the 
USAF  Scientific  Advisory  Board  identified  22  different  missions  that  could  conceivably  be  adequately 
performed  by  UAVs.  How  many  of  these  will  be  fulfilled  remains  to  be  seen;  but  it  is  clear  that  UAVs 
will  become  increasing  more  important. 

Who  should  operate  and  employ  these  aircraft?  Military  personnel?  Officers  or  enlisted?  Pilots? 
DoD  civilians,  contractors?  The  issue  of  selecting  and  training  operators  has  been  a  topic  of  discussion 
among  senior  military  leadership.  What  types  of  skills  are  required?  What  kind  and  how  much  training 
are  required?  Should  there  be  a  specialized  UAV  career  field?  Implications  of  the  answers  to  these 
types  of  personnel  staffing  questions  have  both  short-term  and  long-term  implications.  Currently,  across 
different  military  services,  there  are  vast  differences  in  both  the  types  of  personnel  who  are  selected  to 
become  UAV  operators  (Weeks,  2000)  and  the  training  programs. 

While  opinions  are  varied  and  easy  to  come  by  (Tobin,  1999),  there  is  very  little  empirical 
research  to  guide  the  leadership  in  determining  the  best  policy  for  selection  and  training.  As  a  direct 
result  of  the  1997  Corona  South  four-star  general  officer  summit,  the  Air  Force  Research  Laboratory 
(AFRL)  undertook  a  study  to  determine  the  skills  required  to  operate  the  Predator  UAV— the  only 
operational  UAV  in  the  USAF  inventory  (Hall  &  Tirre,  1998).  The  initial  study  was  a  survey  of 
experienced  Predator  pilots.  This  study  concluded  that  some  manned  flying  experience  was  necessary 
for  pilots  to  be  successful  and  that  this  experience  should  be  roughly  comparable  to  undergraduate 
pilot  training  (UPT).  However,  the  conclusions  of  that  study  were  based  on  opinions.  There  was  a 
desire  to  follow  up  the  opinion  survey  with  a  study  that  contained  some  performance  measurement  data 
addressing  the  issue.  The  study  presented  in  this  paper  is  the  result.  The  purpose  of  this  experiment 
was  to  determine  how  much,  if  any,  flying  experience  is  necessary  to  successfully  operate  the  Predator 
UAV. 


Because  there  are  a  number  of  potential  issues  that  can  be  raised  in  this  context,  it  is  important  to 
state  what  the  intent  of  this  study  was  not,  as  well  as  what  it  was: 

-  It  was  not  the  purpose  to  address  the  issue  of  officer  vs.  enlisted.  It  was  stipulated  at  the  outset 
that  as  a  matter  of  policy  the  USAF  was  not  considering  the  use  of  enlisted  personnel. 

-  It  was  also  not  the  intent  to  include  many  skills  that  may  fall  under  the  rubric  of  “airmanship” 
resulting  from  command  experience  (e.g.,  communication  up  and  down  the  chain  of  command,  tactical 
decision  making).  The  comprehensive  set  of  skills  required  is  far  too  large  to  be  adequately  addressed  in 
one  experiment. 
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-  It  was  not  the  intent  to  conduct  a  study  pertinent  to  all  UAVs.  The  present  study  was  limited  to 
the  Predator  and  the  results  are  intended  for  application  to  the  Predator  system  only.  Other  UAVs  may  be 
highly  automated  with  far  different  missions  and  concept  of  operations  than  Predator.  It  should  be  noted 
that  several  significant  changes  to  the  Predator  (both  system  and  mission)  have  occurred  since  this  study 
was  initiated. 

-  It  was  not  the  intent  to  address  issues  such  as  creation  of  specialized  UAV  career  fields,  training 
syllabus,  tour  length,  or  follow-on  assignments.  For  example,  if  new  graduates  of  T-38  training  were 
selected  to  fly  the  Predator,  would  their  flying  skills  be  maintained  sufficiently  to  receive  a  subsequent 
assignment  to  a  fighter  aircraft?  Issues  such  as  this  require  a  different  study  methodology. 

-  This  study  did  not  attempt  to  identify  the  role  that  motivational  factors  play  in  performance  on 
the  experimental  tasks. 

-  The  Predator  is  crewed  by  a  team  of  operators.  This  experiment  did  not  address  any  “team” 

issues. 


-  Some  changes  from  the  original  research  plan  were  required  due  to  logistics  and  funding  con¬ 
straints.  The  original  plan  included  testing  of  nonpilots  at  the  Air  Force  Academy  (we  used  Air  Force 
Reserve  Officer  Candidate  [ROTC]  students  instead).  It  also  included  a  group  of  students  who  were  plan¬ 
ning  to  enter  Air  Force  career  fields  other  than  flying;  and  it  included  some  pencil-and-paper  ability  testing 
of  some  of  the  groups.  In  our  view,  none  of  these  changes  affected  the  validity  of  the  study. 

It  is  a  fair  characterization  to  say  that  this  experiment  focused  on  the  psychomotor  and  perceptual 
skills  required  to  pilot  the  Predator.  It  is  possible  that  the  results  of  the  study  may  be  used  to  guide 
decisions  regarding  some  of  the  issues  mentioned  above,  but  it  would  have  to  be  done  in  combination 
with  other  data,  information,  and  assumptions. 

Currently,  the  Predator  pilot  is  a  military  pilot  with  experience  in  at  least  one  other  operational 
weapons  system.  Due,  in  part,  to  a  critical  shortage  of  military  pilots  available  for  manned  aircraft,  the 
question  arose  as  to  whether  the  pilot  position  for  the  Predator  needed  to  have  such  extensive  qualifica¬ 
tions.  In  fact,  on  occasion,  an  experienced  weapons  system  operator  or  navigator  with  a  commercial 
instrument  rating  has  been  selected  for  Predator  pilot  training.  While  it  is  certainly  the  case  that  the 
pilot  shortage  is  a  short-term  situation,  the  issue  is  valid  in  any  case.  The  expected  expansion  of  the 
numbers  of  UAVs  requires  that  these  very  issues  be  answered  sooner  rather  than  later.  What  are  the 
most  logical  alternatives? 

Based  on  the  results  of  the  initial  survey,  the  most  likely  candidate  groups  seemed  to  be  UPT 
graduates  or  officers  with  commercial  instrument  pilot  credentials.  Flowever,  it  is  always  wise  to  bound 
an  experiment  with  top,  bottom,  and  intermediate  levels  of  the  critical  variable,  in  this  case,  flying 
experience.  So,  a  group  of  individuals  who  meet  the  physical  and  intellectual  requirements  but  had  no 
flying  experience  would  constitute  the  bottom  of  the  experience  dimension.  In  this  case,  this  level  was 
filled  with  Air  Force  ROTC  cadets  who  had  no  previous  flying  experience.  The  top  level  of  experience 
was  filled  by  current  mission-qualified  Predator  pilots. 

Given  time  and  resources,  most  people  who  meet  the  physical  and  intellectual  requirements  of  an 
Air  Force  officer  could  be  trained  to  operate  any  system.  However,  the  current  initial  qualification  train- 
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ing  (IQT)  for  the  Predator  leverages  heavily  from  an  assumed  entry  level  of  knowledge  and  experience  in 
the  operational  military  aviation  system.  Any  change  from  the  current  selection  policy  will  undoubtedly 
have  ramifications  for  the  design  of  the  training  program.  Therefore,  it  was  also  the  intent  of  this  study  to 
design  the  experiment  so  that  it  has  relevance  to  the  training  issues  as  well  as  selection  issues. 


OBJECTIVE 

The  objective  of  this  study  is  to  provide  empirical  evidence  on  the  question  of  how  quickly  and 
how  well  Predator  piloting  skills  can  be  learned  by  various  groups  of  pilots  (and  future  pilots)  who  differ 
in  amount  and  type  of  prior  flight  training  and  experience.  The  study  addresses  the  set  of  skills  required 
to  control  a  Predator  accurately  during  difficult  maneuvers.  It  also  provides  some  data  regarding  the 
skills  required  to  stay  oriented  while  maneuvering  to  accomplish  a  specific  reconnaissance  objective. 
However  it  was  not  designed  to  address  other  skills  that  are  important  components  of  a  well-rounded 
operator,  for  example,  combat  airspace  management,  communicating  information  to  a  command  chain, 
and  other  officership  issues. 


APPROACH 


Participants 

Participants  were  recruited  from  several  groups  of  pilots  (or  students  wishing  to  become  pilots). 
Five  of  these  groups  represented  populations  from  which  future  Predator  UAV  pilots  could  conceivably 
be  drawn  (i.e.,  candidate  groups);  two  others  were  reference  groups  at  the  extreme  ends  of  the  flying 
experience  spectrum. 

Experienced  Predator  Pilots.  Six  experienced  pilots  participated  at  their  operational  location, 
Indian  Springs  Air  Force  Auxiliary  Field  (AFAF),  NV.  Their  ages  ranged  from  29-43  years;  all  were  men 
with  the  rank  of  captain.  Flying  experience  ranged  from  1,680  to  2,942  hours,  of  which  between  80  and 
340  were  Predator  hours.  All  had  at  least  one  tour  of  duty  in  another  operational  aircraft.  Five  came 
from  wide  body  tanker,  transport,  bomber  (TTB)  aircraft  assignments;  one  came  from  fighters. 

Predator  Selectees.  Twenty-four  Predator  selectees  were  scheduled  for  data  collection  during 
their  first  week  assigned  to  Indian  Springs  AFAF,  before  any  Predator  training  had  begun.  Data  from 
two  selectees  were  not  used  because  it  was  incomplete  due  to  scheduling  or  other  logistics  problems. 
Navigators  with  commercial  instrument  ratings  are  currently  part  of  the  Air  Force  selection  pool  for 
Predator  UAV  pilot  duty,  but  were  not  originally  a  planned  group  in  our  study.  Since  this  population 
was  available  at  Indian  Springs  during  the  study,  we  included  navigator  selectees  in  hopes  of  obtaining 
a  navigator  group  large  enough  to  compare  with  other  groups.  However,  only  four  navigators  were 
obtained— an  insufficient  number  to  include  as  a  separate  group  in  the  study. 

The  remaining  18  pilots,  17  men  and  1  woman,  formed  the  Selectee  group  used  here.  Their 
ages  ranged  from  26-43  years.  The  distribution  of  ranks  was:  1  lieutenant,  14  captains,  1  major,  1 
lieutenant  colonel-select,  and  1  lieutenant  colonel.  Flight  experience  ranged  from  417  to  3,010  hours. 
All  had  at  least  one  tour  of  duty  in  an  operational  aircraft.  Ten  came  from  TTB  aircraft  assignments; 
eight  came  from  fighters. 
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T-38  Graduates.  Eighteen  T-38  students  were  tested  at  their  training  site  (Columbus  Air  Force 
Base,  MS)  after  successful  completion  of  all  training,  just  before  official  graduation  ceremonies.  Data 
from  three  of  these  were  not  used  because  it  was  incomplete.  The  remaining  13  men  and  2  women 
ranged  in  age  from  23-29  years;  13  were  lieutenants,  two  were  captains.  Students  completing  the 
T-37/T-38  UPT  program  finish  with  195-215  total  hours. 

T-l  Graduates.  Nineteen  T-l  students  were  tested  at  their  training  site  (Columbus  Air  Force 
Base,  MS)  after  successful  completion  of  all  training,  just  before  official  graduation  ceremonies.  Data 
from  three  of  these  were  not  used  because  it  was  incomplete.  The  remaining  14  men  and  2  women 
ranged  in  age  from  23-28  years;  14  were  lieutenants,  two  were  captains.  Students  completing  the 
T-37/T-1  UPT  program  finish  with  195-215  total  hours. 

Civilian  Instrument  Pilots.  Sixteen  students  who  recently  completed  requirements  for  the 
civilian  single-engine  instrument  (Course  FA25 1)  were  tested  at  their  training  site  (ERAU,  Prescott,  AZ). 
Data  from  one  of  these  were  not  used  because  it  was  incomplete.  The  remaining  15  pilots  ranged  in  age 
from  20-31  years;  all  were  men.  Flight  experience  ranged  from  120  to  177  hours,  typically  in  Cessna 
(172)  and  Beechcraft  (Be  76)  aircraft. 

Civilian  Private  Pilots.  Seventeen  students  who  recently  completed  requirements  for  the  civilian 
private  pilot  license  were  tested  at  their  training  site  (ERAU  or  Williams  Gateway  Airport,  Mesa,  AZ). 
Data  from  four  of  these  were  not  used  because  it  was  incomplete.  The  remaining  13  pilots  (10  from 
ERAU,  3  from  Williams)  ranged  in  age  from  18-25  years;  all  were  men.  Flight  experience  ranged  from 
45  to  80  hours,  typically  in  a  Cessna  172  aircraft. 

ROTC  Students  (nonpilots).  Sixteen  Air  Force  ROTC  students  were  tested  at  ERAU.  They 
ranged  in  age  from  19-22  years;  all  were  men.  None  had  any  flight  experience,  but  all  intended  to  go 
into  Air  Force  pilot  training. 

Participant  Compensation.  The  Predator  pilots,  Predator  selectees,  T-38  graduates,  and  T-l 
graduates  all  participated  in  this  study  as  part  of  their  normal  Air  Force  duty  day.  Participants  from 
the  three  civilian  groups — instrument  pilots,  private  pilots,  and  ROTC  students — were  paid  $15  an  hour 
for  their  participation.  It  is  important  to  bear  in  mind  that  these  differences  in  the  way  the  participants 
were  compensated,  together  with  other  potential  sources  of  motivational  differences  between  groups, 
could  have  affected  observed  group  differences,  although  there  were  no  obvious  indicators  of  such 
effects  in  the  data. 

Although  the  participant  groups  differ  in  flying  training,  experience,  and  other  factors,  they  share 
two  important  characteristics:  (a)  almost  all  participants  either  have  a  bachelors  (or  higher)  degree, 
or  are  currently  enrolled  in  a  program  to  obtain  one;  and  (b)  all  of  them  share  a  strong  interest  in  a 
career  as  a  pilot. 


Study  Procedures 

As  an  extension  to  an  ongoing  research  project,  Air  Combat  Command  (ACC)  requested  AFRF’s 
Warfighter  Training  Research  Division  (AFRF/HEA)  to  develop  RQ-1 A  Predator  simulators  for  the  1 1th 
and  15th  Reconnaissance  Squadrons.  Several  training  simulation  systems  were  developed  for  AFRF  by 
Parker  International  and  the  ACC  Training  System  Center  (TSC)  Squadron  Detachment  1  at  Fuke  AFB 
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Figure  1.  Predator 
UAV  simulation. 


AZ  and  are  currently  in  use  by  operational  Predator  pilots  at  Indian  Springs  AFAF,  NV.  Just  before  data 
collection  commenced,  RQ-1A  system  #4  was  the  most  current,  so  that  simulation  environment  was  the 
basis  for  the  tasks  given  to  all  participants  in  the  current  study  (Figure  1).  The  simulation  environment 
was  modified  to  allow  the  measurement  of  participants’  performance  while  flying  three  tasks — Basic 
Maneuvering,  Landing,  and  Reconnaissance.  The  simulation  was  run  on  a  dual-Pentium  PC,  with  an 
additional  PC  networked  to  it  for  experimental  control  and  presentation  of  tutorial  materials.  Participants 
controlled  the  simulated  Predator  with  the  same  primary  controls  used  on  the  actual  system— stick, 
throttle  and  rudder  pedals  (Note:  though  System  #4  did  have  a  control  for  propeller  pitch,  the  plane 
did  not  have  an  adjustable  propeller,  so  we  did  not  provide  the  control.  We  also  did  not  provide  the  flap 
lever  control  device,  because  Predator  pilots  stated  they  would  not  adjust  the  flaps  during  any  of  our  three 
tasks,  either  in  simulation  or  with  the  actual  plane). 


Figure  2  shows  the  equipment  used  for  the  study.  Tasks  were  presented  on  two  side-by-side 
19”  color  monitors,  the  left  displaying  the  exact  System  4  “Heads-Up  Display  (HUD)”  instrumentation 
overlaid  on  either  a  pure  black  screen  (Basic  Maneuvering  Task)  or  a  computer-generated  camera  view 
(Landing  and  Reconnaissance  Tasks),  and  the  right  monitor  displaying  the  tracker  map  display  and 
associated  symbology.  A  panel  on  the  right  of  the  tracker  map  presented  additional  information  relevant 
to  the  task.  Some  of  this  information  would  normally  be  available  from  the  variable  information  tables 
(VIT)  in  the  actual  Predator  ground  control  station  (GCS),  while  other  information  was  pertinent  only 
to  our  particular  research  tasks.  A  detailed  description  of  each  of  the  tasks  used  in  the  study,  including 
pictures  of  the  displays  and  description  of  the  equipment,  is  given  in  Appendix  A;  software  description 
is  given  in  Appendix  B. 


The  general  sequence  of  events  in  the  study  was  as  follows: 


After  being  briefed  on  the  study  purpose  and  signing  consent  forms,  participants  viewed  a 
self-paced,  computer-based  tutorial  that  used  text,  graphs,  pictures,  overlays,  videos,  and  animations  to 
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Figure  2.  Predator 
Synthetic  Task 
Environment 
equipment  setup. 


convey  declarative  and  procedural  knowledge  about  the  Predator  simulation  and  the  task  to  be  flown. 
After  finishing  the  Basic  Maneuvering  tutorial,  the  participant  received  a  written  test  on  the  material.  For 
incorrect  answers,  the  participant  was  asked  to  review  the  tutorial  to  determine  the  correct  answers.  After 
assuring  that  the  participant  understood  the  tutorial  material  (i.e.,  a  written  test  score  of  100%),  he  or  she 
was  shown  the  displays  and  controls  for  the  Basic  Maneuvering  Task,  given  written  reference  sheets  for 
the  task,  and  walked  through  a  practice  trial.  Feedback  was  presented  using  graphical  and  text  displays 
of  performance  measures.  Then  the  participant  repeated  the  Basic  Maneuvering  Task  with  computerized 
feedback  until  performance  on  all  aspects  of  it  reached  criterion  levels.  The  same  general  procedure 
was  used  for  the  other  two  tasks — first  a  computer-based  tutorial,  then  a  test,  then  a  walk-through 
and  practice  trial,  and  then  performance  with  feedback  of  the  task  itself.  Flowever,  the  Reconnaissance 
Task  was  run  for  a  fixed  number  of  trials  (30,  including  practice)  instead  of  running  until  criterion 
performance  was  reached.  Following  the  reconnaissance  trials,  the  participant  filled  out  a  demographic 
questionnaire  and  was  debriefed.  Total  time  required  per  participant  was  12  to  30  hours,  depending  on 
how  long  a  participant  studied  tutorial  preparatory  information  and  trial  feedback  graphs,  and  how  many 
attempts  were  required  to  pass  the  Basic  Maneuvering  and  Landing  Tasks. 


Research  Tasks 

Task  Validity.  Prior  to  the  design  of  the  research  tasks,  extensive  structured  interviews  with 
experienced  Predator  operators  were  conducted  at  Indian  Springs  AFAF.  These  interviews  were  part  of  a 
cognitive  task  analysis  designed  to  yield  information  about  which  parts  of  the  Predator  mission  pose  the 
greatest  difficulties  for  the  operators.  Following  this  analysis,  three  tasks  were  designed  to  tap  different 
aspects  of  Predator  UAV  piloting  skill.  The  details  of  each  task — breakdown  of  the  task  into  elements  or 
segments,  design  of  scenarios,  selection  of  performance  measures,  and  setting  of  pass/fail  criteria — were 
worked  out  with  assistance  from  two  former  Predator  UAV  pilots  under  subcontract  to  AFRL/HEA. 
Fidelity  of  the  handling  characteristics  of  the  underlying  simulation  was  established  through  a  long 
process  of  iterative  testing  by  experienced  pilots  at  Indian  Springs  AFAF. 

Task  Sequence.  Since  only  the  Predator  pilot  group  was  familiar  with  any  aspects  of  the 
Predator  UAV,  the  first  task  all  participants  encountered — the  Basic  Maneuvering  Task — was  designed 
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to  bring  all  participants  up  to  a  common  minimum  proficiency  level  on  stick-and-rudder  skills  operating 
the  Predator  UAV  at  altitude.  Therefore,  the  Basic  Maneuvering  tutorial  covered  (a)  basic  declarative 
and  procedural  flight  knowledge  (for  the  benefit  of  the  non-pilot  ROTC  group),  (b)  unusual  UAV 
characteristics  (e.g.,  control  transmission  delay,  stability  augmentation  system),  (c)  other  platform- 
specific  information,  such  as  the  unusual  HUD,  and  (d)  instructional  information  specific  to  the  Basic 
Maneuvering  Task.  Successful  completion  of  the  Basic  Maneuvering  tutorial  and  task  demonstrated  (a) 
memorization  of  gauge  location  and  function,  (b)  familiarity  with  the  aircraft  handling  characteristics 
and  their  interactions,  and  (c)  ability  to  fly  the  Predator  using  instruments  only.  Finally,  because 
successful  completion  of  the  Basic  Maneuvering  Task  brought  all  participants  to  comparable  competency 
levels  on  the  above,  differences  observed  in  the  following  tasks  (Landing  and  Reconnaissance,  in  that 
order)  would  be  attributable  to  more  complex  skills,  rather  than  basic  learning  of  the  instruments  and 
general  handling  characteristics. 

Basic  Maneuvering  Task.  The  Basic  Maneuvering  Task,  similar  to  some  civilian  training  tasks 
(e.g.,  standard  rate  turn)  and  derived  from  the  task  used  in  Wickens,  Bellenkes  and  Kramer  (1995), 
required  participants  to  execute  maneuvers  in  which  the  goal  was  to  change  one  or  more  axis  of  flight- 
airspeed,  altitude  and/or  heading— at  a  precise,  constant  rate  of  change  during  a  60-90  second  trial.  The 
criterion  performance  measure  on  each  trial  was  deviation  from  the  desired  rate  of  change  for  each 
parameter,  specifically  the  root-mean-square  (RMS)  deviation  from  the  instructed  rate  of  change  for 
each  of  the  three  parameters.  There  were  seven  basic  maneuvering  subtasks  (called  “segments”).  The 
first  three  segments  required  the  participant  to  change  just  one  parameter  while  keeping  the  other  two 
parameters  constant.  The  first  was  a  constant-rate  reduction  in  airspeed,  the  second  was  a  constant-rate 
turn,  and  the  third  was  a  constant-rate  change  in  altitude.  The  next  three  trials  required  changing 
two  parameters  simultaneously  (for  example,  a  constant  rate  turn  while  decreasing  airspeed).  The  final 
segment  required  changing  all  three  parameters.  The  participant  flew  each  segment  repeatedly  until 
achieving  RMS  criterion  simultaneously  on  all  three  parameters,  and  then  was  moved  on  to  the  next 
segment. 

Landing  Task.  The  Landing  Task  preserved  much  of  the  difficulty  of  landing  a  real  Predator. 
Landing  is  considered  the  most  difficult  operational  task  Predator  pilots  face.  The  design  characteristics 
of  the  UAV  make  it  very  responsive  to  control  surface  manipulations  and  sensitive  to  winds.  Predator 
pilots  cannot  rely  on  vestibular  cues  during  flare  and  touchdown  (e.g.,  a  sinking  feeling),  they  have  less 
optical  flow  information  available,  especially  in  the  far  periphery  due  to  the  small,  30-degree  field-of- 
view  camera  (e.g.,  ground  and  bushes  rising  on  the  sides  of  the  runway  just  before  touchdown),  and  they 
must  also  deal  with  a  line-of-sight  control  transmission  delay. 

The  Landing  Task  used  in  this  study  incorporated  all  the  above  challenges  and  was  designed  to  be 
the  most  difficult  test  of  Predator  stick-and-rudder  skill.  Successful  completion  of  the  Landing  tutorial 
and  task  demonstrated  that  the  participant  was,  to  use  piloting  terminology,  able  to  “fly  several  seconds 
ahead  of  the  aircraft,”  resulting  in  high  levels  of  stick-and-rudder  skill  and  an  ability  to  predict  how 
the  plane  would  react  several  seconds  after  an  action.  (As  a  further  experimental  benefit,  once  this 
high  level  of  stick-and-rudder  skill  was  achieved  on  the  Landing  Task  by  all  participants,  any  group 
differences  observed  on  the  subsequent  Reconnaissance  Task  would  be  due  largely  to  other  skills  beyond 
stick  and  rudder.) 

At  the  beginning  of  a  landing  attempt,  the  participant  is  initialized  on  the  downwind  leg  of  an 
approach  pattern  commonly  used  at  Indian  Springs  and  must  fly  the  rest  of  the  pattern  until  touchdown 
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Figure  3.  The  tracker  map 
is  displayed  on  the  right- 
hand  monitor.  This  exam¬ 
ple  shows  a  plane  on  final 
approach.  The  participant 
sees  only  the  map  and 
plane  icon;  we  have  super¬ 
imposed  the  approach  path 
(red  line),  turn  points  (red 
dots)  and  altitude  gate 
positions  (black  lines)  for 
illustrative  purposes  only. 


(Figure  3).  Landings  without  wind  were  flown  repeatedly  until  criterion  was  reached  simultaneously 
on  thirteen  criterion  measures  of  performance  (i.e.,  a  perfect  landing).  These  measures  assessed  error  in 
approach  altitude  gates  and  groundtrack  pattern,  glideslope,  and  touchdown  parameters  (the  complete 
list  is  in  Appendix  A.)  Once  a  participant  reached  criterion  on  the  no-wind  landing,  trials  landing  with 
crosswind  were  presented.  (Interviews  with  experienced  Predator  pilots  revealed  crosswind  landings 
as  the  most  difficult  part  of  any  mission.)  Before  commencing  with  crosswinds,  the  participants 
viewed  two  instructional  simulated  Predator  crosswind  landing  videos  and  were  provided  a  crosswind 
declarative  and  procedural  instruction  sheet.  Wind  was  always  constant  at  13  knots  at  all  altitudes  for 
all  crosswind  trials,  but  could  originate  from  any  of  four  directions— 0  deg,  180  deg,  45  deg,  or  135  deg 
for  a  given  attempt.  Participants  were  heading  at  approximately  90  deg  on  final.  As  with  the  no-wind 
trials,  the  participant  flew  repeated  landings  until  a  perfect  landing  was  achieved,  passing  all  13  criteria 
simultaneously  on  a  single  landing  attempt. 

Reconnaissance  Task.  The  third  and  final  task  involved  the  Predator’s  mission—  reconnaissance. 
On  occasion,  the  Predator  is  tasked  to  obtain  electro-optical  imagery  of  a  target  under  conditions  of 
almost  complete  cloud  cover.  Under  these  conditions  it  is  important  to  maximize  the  time  during  which 
an  unobstructed  view  of  the  target  is  available  so  that  all  of  the  detailed  reconnaissance  objectives  can 
be  satisfied.  Though  this  tasking  is  somewhat  infrequent,  it  was  the  task  deemed  most  difficult  for  the 
pilot  on  a  Predator  reconnaissance  mission.  A  simulation  task  reflecting  this  mission  is  therefore  the  best 
opportunity  to  reveal  performance  differences  between  the  groups  in  this  study.  The  Reconnaissance 
Task  requires  the  pilot  to  plan,  maneuver  in  a  target  area,  and  maintain  orientation  (as  opposed  to  the 
aircraft  handling  skill  tested  in  the  first  two  tasks).  Initial  interviews  and  task  analysis  indicated  that 
difficult  scenarios  in  this  task  would  require  substantial  aviation-related  spatial  reasoning  skills. 

The  Reconnaissance  Task  required  the  participant  to  fly  30  scenarios  in  which  they  were  to  obtain 
as  much  time-on-target  as  possible  within  a  10-minute  scenario  time  limit.  To  perform  each  scenario, 
the  participant  must  position  the  aircraft  so  that  the  sensor  camera  points  toward  a  target  through  a  small 
break  in  a  cloud  layer  (Figure  4).  Restrictions  similar  to  those  encountered  in  an  operational  mission  are 
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Figure  4.  Schematic 
drawing  of  the  Recon¬ 
naissance  Task. 


incorporated  in  the  task.  For  example,  participants  are  penalized  for  entering  no-fly  zones,  violating 
altitude  restrictions,  stalling  the  aircraft,  and  exceeding  bank  and  depression  angle  limits  while  viewing 
the  target.  Scenarios  differed  in  wind  direction  (10-330  deg.),  wind  speed  (0,  15,  and  30  knots),  and 
placement  of  no-fly  zones  near  the  area  from  which  the  target  can  be  viewed. 

To  create  our  single-person  pilot  Reconnaissance  Task,  we  always  initialized  the  pilot  in  the 
immediate  target  area  with  targeting  mode  on  (with  the  precise  coordinates  of  the  target),  thereby 
focusing  on  the  pilot-only  skills  involved  in  this  type  of  reconnaissance  mission  (see  Martin,  Lyon,  & 
Schreiber,  1998).  At  any  time,  the  pilot  could  press  a  key  to  switch  from  nose  camera  view  (to  search 
for  the  cloud  hole)  to  payload  camera  view  (to  accrue  time  on  target).  Fligh  time  on  target  on  the 
Reconnaissance  Task  demonstrates  good  planning,  good  understanding  of  the  three-dimensional  problem 
space,  and  proper  execution. 


Data  Analysis 

Each  of  the  three  synthetic  tasks  has  a  primary  measure  to  best  indicate  successful  performance. 
For  the  Basic  Maneuvering  and  Landing  Tasks,  this  measure  is  the  number  of  trials  required  to  reach 
criterion  performance  (i.e.,  successful  completion).  For  the  Reconnaissance  Task,  it  is  total  time  on 
target  during  a  scenario.  Each  task  also  has  numerous  built-in  subsidiary  performance  measures  that 
provided  further  information  about  the  nature  of  differences  between  groups. 

Mean  performance  for  each  participant  on  each  task  was  obtained  for  each  measure.  Group  mean 
plots  of  selected  measures  (with  error  bars  indicating  plus-or-minus  one  standard  error)  are  presented  in 
the  results  section.  Significance  levels  for  the  overall  effect  of  pilot  group  were  obtained  using  SPSS  via 
a  one-way  analysis  of  variance  (ANOVA).  Given  a  significant  overall  effect,  post-hoc  tests  for  pairwise 
group  comparisons  (corrected  for  the  number  of  comparisons  within  the  ANOVA)  were  conducted. 


9 


In  all  three  tasks,  all  trials  after  the  initial  walk-through  trial  were  included  in  the  analyses. 
However,  in  the  Landing  Task,  the  more  detailed  subsequent  analyses  excluded  trials  in  which  any  of 
the  following  occurred  in  a  given  landing  attempt:  (a)  the  participant  flew  more  than  50%  of  the  way 
down  the  runway  without  landing;  (b)  the  participant  purposefully  nosed  the  plane  into  the  ground  early 
in  the  trial  to  avoid  having  to  complete  an  already  spoiled  landing;  or  (c)  the  participant  pressed  the 
abort  key  to  indicate  a  “go-around”  is  necessary.  Combined,  these  trials  represented  2.8%  of  the  total 
7,124  landing  trials  examined. 

For  most  analyses  of  the  landing  data,  no-wind  and  crosswind  trials  were  combined  because  a 
preliminary  examination  revealed  no  differential  effect  of  the  presence  of  wind  on  group  performance. 
Wind  simply  increased  the  difficulty  for  all  groups.  However,  for  a  few  parameters,  performance  on 
no-wind  trials  was  consistently  better  than  criterion  level  for  all  groups,  whereas  potentially  interesting 
differences  between  groups  emerged  only  on  trials  with  wind.  For  those  parameters,  wind  and  no-wind 
trials  are  graphed  and  analyzed  separately. 


RESULTS 

The  results  of  the  study  will  be  described  as  follows.  First,  the  groups  will  be  compared 
on  the  two  primary  overall  measures,  i.e.:  (a)  the  combined  total  number  of  training  trials  required  to 
achieve  criterion  performance  through  the  crosswind  landings  (i.e.,  basic  maneuvering  attempts  plus  no¬ 
wind  landing  attempts  plus  crosswind  landing  attempts);  and  (b)  total  target  data  collection  time  during 
the  Reconnaissance  Task.  Second,  performance  on  each  of  the  three  tasks  will  be  analyzed  separately  so 
that  the  determinants  of  overall  performance  can  be  understood.  Finally,  some  of  the  data  from  the  exit 
demographic  questionnaire  will  be  discussed. 


Overall  Performance 

Total  Basic  Maneuvering  and  Landing  Trials.  Figure  5  shows  the  mean  total  number  of 
training  trials  required  to  reach  criterion  on  the  combined  Basic  Maneuvering  and  Landing  Tasks.  This 
is  the  best  estimate  of  the  relative  amount  of  practice  required  to  bring  groups  up  to  a  criterion  Predator 
UAV  stick-and-rudder  proficiency  level.  The  overall  effect  of  pilot  group  was  highly  significant  (p 
<  0.001).  Predator  pilots  required  the  fewest  training  trials  (50.2),  whereas  nonpilot  ROTC  students 
required  the  most  (160.8)— more  than  three  times  the  number  required  by  pilots  and  significantly  more 
than  every  other  group  except  T-l  graduates.  T-l  graduates  required  significantly  more  trials  (123.7) 
than  Predator  selectees  (84.6,  p  =  0.039)  and  civilian  instrument  pilots  (84.5,  p  =  0.047).  The  other 
groups  of  pilots  without  Predator-specific  experience  (T-38  graduates  and  private  pilots)  did  not  differ 
significantly  from  each  other  or  from  either  Predator  selectees  or  T-l  graduates. 

These  results  show  that  previous  flying  experience  can  substantially  reduce  the  number  of  trials 
required  to  become  proficient  in  basic  maneuvering,  and  no-wind  and  crosswind  landing.  It  should 
be  noted  that,  for  the  Landing  Task,  the  number  of  trials  to  criterion  may  be  an  underestimate  of 
the  true  number  of  trials  to  reach  criterion  performance.  Because  the  ROTC  students  were  nonpilots, 
much  learning  needed  to  take  place  for  them  to  complete  the  landings,  especially  crosswind  landings. 
We  determined  that  any  participant  who  could  not  pass  the  landing  trials  within  200  attempts  needed 
coaching  to  fully  understand  the  complex  stick-and-rudder  skills  necessary  to  land  the  Predator  UAV  in 
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Figure  5.  Total  number 
of  trials  required  to 
achieve  criterion  per¬ 
formance  on  Basic 
Maneuvering,  No-Wind 
Landing  and  Crosswind 
Landing.  Points  are 
means  of  total  number 
of  trials  for  individuals 
within  each  group. 


crosswind.  While  no  pilot  reached  200  attempts,  three  of  the  ROTC  students  reached  this  instructional 
intervention  point.  Without  this  intervention,  those  students  might  have  required  many  more  trials. 
Further,  as  the  task-by-task  analyses  presented  below  will  show,  passing  our  strict  criteria  does  not  assure 
that  pilots  are  equally  skilled.  So  the  differences  between  groups  in  number  of  trials  should  be  viewed 
only  in  relative  terms.  The  number  of  trials  required  to  bring  nonpilots  to  a  skill  level  truly  equal  to 
Predator  selectees  on  crosswind  landings  would  probably  require  even  more  than  the  76-trial  difference 
observed  here.  Nevertheless,  even  this  difference,  if  translated  to  landings  with  real  aircraft,  represents  a 
considerable  investment  in  training  time  and  resources. 

Reconnaissance  Task  Time  on  Target.  Figure  6  shows  the  mean  time  on  target  during  each 
scenario  of  the  Reconnaissance  Task.  The  effect  of  pilot  group  was  highly  significant  (p  <  0.001). 
Nonpilots  logged  significantly  less  time  on  target  (90.2  sec)  than  Predator  pilots  (130.7  sec,  p  <  0.001), 
Predator  selectees  (125.2  sec,  p  <  0.001),  T-38  graduates  (126.0  sec,  p  <  0.001)  and  T-l  graduates  (1 1 1.8 
sec,  p  =  0.017).  Predator  pilots,  Predator  selectees  and  T-38  graduates  had  significantly  more  time  on 
target  than  any  civilian  group  (p’s  from  <  0.001  to  0.043). 

These  results  show  that  even  after  the  groups  are  all  brought  to  a  criterion  level  of  stick-and- 
rudder  ability  to  handle  the  simulated  Predator  aircraft,  different  types  and  amounts  of  flying  experience 
still  determine  success  on  the  Reconnaissance  Task.  Our  analysis  of  Reconnaissance  Task  performance 
will  provide  evidence  that  this  effect  of  pilot  group  is  not  due  solely  to  residual  differences  in  stick-and- 
rudder  skills,  but  rather  to  some  higher  order  cognitive  skills  required  for  maneuvering  and  maintaining 
orientation. 

Summary.  Both  the  overall  trials-to-criterion  data  and  the  reconnaissance  time-on-target  data 


11 


140 


130 


eo 

d 

D 

O 

C/5 

!h 

D 

Oh 

Id 

OD 

H 

a 

o 

D 

a 

H 

§ 

D 


120 


110 


100 


90 


Figure  6.  Time  that 
the  sensor  camera  was 
viewing  the  target 
through  the  cloud  hole, 
and  the  aircraft  was 
not  in  bank,  depression 
angle,  stall,  altitude,  or 
no-fly-zone  violations. 
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suggest  that,  although  the  effect  of  pilot  group  (and  therefore  flying  experience)  is  primary  and  highly 
significant,  amount  of  prior  flying  time  is  not  the  only  determinant  of  performance.  Some  groups  do 
worse  and  others  do  better  than  would  be  expected  on  the  basis  of  flying  hours  alone.  Our  analysis  of 
performance  on  individual  tasks  will  suggest  some  reasons  for  this. 


Details  of  Performance  on  Individual  Tasks 


Basic  Maneuvering  Task 

Figure  7  shows  the  mean  total  number  of  training  trials  required  to  pass  all  seven  segments 
of  the  Basic  Maneuvering  Task.  Flere  the  effect  of  pilot  group  does  not  reach  significance  (p  = 

0.074).  This  task  is  not  nearly  as  difficult  as  the  Landing  Task.  It  generally  requires  many  fewer 
trials  to  pass.  However,  it  was  necessary  to  include  the  task  so  that  nonpilots  could  leam  to  control 
the  simulated  aircraft  and  other  non-Predator  pilots  could  become  familiar  with  the  gauges  and  handling 
characteristics. 

Although  groups  did  not  differ  significantly  in  number  of  trials  to  criterion,  they  differed  dramati¬ 
cally  in  more  fine-grained  measures  of  aircraft  control.  Figure  8  shows  the  normalized  total  RMS  error 
in  performing  all  seven  segments  of  the  task.  This  score  represents  how  closely  each  pilot  was  able  to 
keep  the  aircraft  to  the  desired  airspeed,  heading,  and  altitude  simultaneously,  a  measure  of  total  flying 
deviation  error  on  all  three  parameters.  To  compute  this  score,  mean  RMS  error  for  each  subject  in 
airspeed,  heading,  and  altitude  was  calculated.  Then  the  error  values  on  each  of  these  parameters  were 
normalized  so  they  could  be  added  together  as  z-scores,  thus  assuring  that  none  of  them  were  improperly 
weighted  in  the  average.  The  resulting  total  flying  deviation  error  z-scores  show  a  highly  significant 
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Figure  7.  Number  of  trials 
required  to  achieve  criterion 
performance  on  the  Basic 
Maneuvering  task. 


effect  of  pilot  group  (p  =  0.003).  Figures  9,  10,  and  1 1  show  the  raw  RMS  error  scores.  Clearly,  error  in 
airspeed  and  heading  were  the  biggest  contributors  to  differences  between  groups  in  overall  error. 

The  pilot  groups  also  differed  in  the  variability  and  smoothness  with  which  the  simulated  Preda¬ 
tor  was  controlled.  We  computed  two  scores  that  are  related  to  these  characteristics.  The  variability 
score  is  called  the  standard  variance  composite,  which  was  obtained  by  normalizing  the  variances  of 
(absolute)  pitch,  bank,  and  rudder  control  values  and  summing  them.  The  standard  variance  composite 
reflects  the  extent  to  which  the  participant  made  numerous  control  inputs.  That  is,  frequent  control 
inputs  tend  to  produce  a  higher  standard  variance  composite  score.  Similar  to  the  standard  variance 
composite,  the  rate  variance  composite  reflects  the  smoothness  of  control  rate  inputs  and  is  the  normal¬ 
ized  sum  of  variances  in  pitch,  roll,  and  yaw  rates.  Large,  quick  control  inputs  tend  to  produce  high  rate 
variance  composite  scores.  Figures  12  and  13  show  these  scores.  For  both  variables,  the  effect  of  pilot 
group  was  highly  significant  (standard  variance,  p  =  0.002;  rate  variance,  p  <  0.001).  Predator  pilots 
exhibited  significantly  less  variable  and  smoother  performance  than  the  Predator  pilot  selectees  (standard 
variance,  p  =  0.046;  rate  variance,  p  =  0.003),  and  the  two  civilian  groups  exhibited  significantly  lower 
variability  and  better  smoothness  scores  than  the  two  military  pilot  groups  with  comparable  numbers 
of  flight  hours,  i.e.,  T-38  and  T-l  graduate  (p-values  from  .001  to  .028).  This  advantage  for  the  civilian 
groups  may  be  due  to  familiarity  and  recency  of  flying  a  small  propeller  aircraft. 


Landing  Task 

Figure  14  shows  the  mean  total  number  of  training  trials  required  to  pass  both  the  no-wind  and 
crosswind  landing  trials.  The  overall  effect  of  pilot  group  was  highly  significant  (p  <  .001).  ROTC 
students  required  significantly  more  trials  than  any  other  group.  Predator  pilots  required  significantly 
fewer  trials  than  ROTC,  private  pilots,  or  T- 1  graduates.  Predator  selectees  and  instrument  pilots  also 
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Figure  8.  Sum  of  normalized 
RMS  error  scores  on  altitude, 
airspeed  and  heading  parame¬ 
ters  during  performance  of  the 
Basic  Maneuvering  task. 
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Figure  9.  RMS  error  on 
the  airspeed  parameter  in  the 
Basic  Maneuvering  Task. 
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Figure  10.  RMS  error 
on  the  heading  parameter 
in  the  Basic  Maneuvering 
task. 
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Figure  1 1 .  RMS  error 
on  the  altitude  parameter 
in  the  Basic  Maneuvering 
task. 
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Figure  12.  Standard  vari¬ 
ance  composite  score  for 
the  Basic  Maneuvering  task. 
Score  is  the  normalized  sum 
of  variances  in  pitch,  bank 
and  rudder  control.  Higher 
scores  represent  more  varia¬ 
tion  in  the  size  of  stick  and 
rudder  inputs. 
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Figure  13.  Standard  rate 
variance  composite  score  for 
the  Basic  Maneuvering  task. 
Score  is  the  normalized  sum 
of  variances  in  pitch  rate,  roll 
rate  and  yaw  rate.  Higher 
scores  represent  more  varia¬ 
tion  in  the  rate  of  stick  and 
rudder  inputs. 
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Figure  14.  Total  Number 
of  Landing  Trials  to  Reach 
Criterion. 


required  fewer  trials  than  T-l  graduates  (p  =  0.043  and  p  =  0.047  respectively).  It  should  be  noted  that 
to  pass  either  the  no-wind  or  the  crosswind  landing,  the  participant  had  to  perform  a  perfect  landing  on 
all  13  required  criteria  simultaneously.  This  “perfect”  landing  criteria  was  a  higher  standard  than  even 
the  experienced  Predator  pilots  expected,  hence  even  they  required  multiple  attempts  to  achieve  criterion. 
Had  the  criterion  been  lower,  fewer  attempts  would  have  been  necessary  to  pass,  and  our  confidence  in 
the  task’s  ability  to  assess  difficult  cross-wind  landings  would  have  been  reduced. 


Dangerous  Landings.  Perhaps  as  important  as  the  total  trials  to  reach  criterion  performance  are 
differences  between  groups  on  the  number  of  potentially  dangerous  landings.  A  dangerous  landing  could 
be  the  result  of  a  number  of  factors,  such  as  nose-low,  nose-high,  wing-low,  high  lateral  velocity,  large 
distance  off  centerline,  or  hard  touchdown.  Here  we  report  only  those  dimensions  for  which  a  sizable 
sample  of  dangerous  landings  was  observed.  Even  though  the  database  for  analysis  is  substantially 
reduced  to  examine  only  these  infrequent  events,  meaningful  trends  and  significant  results  were  still 
found  between  pilot  groups. 


Nose  Wheel  Landings.  Figure  15  shows  the  significant  (p  <  .04)  effect  of  pilot  group  on  the 
proportion  of  nose  wheel  landings,  defined  as  landings  in  which  the  simulated  aircraft’s  pitch  was  less 
than  zero  degrees  at  touchdown.  This  type  of  landing  was  usually  the  result  of  either  a  late/nonexistent 
flare  or  an  early  flare  followed  by  a  wing  stall  and  subsequent  “porpoise”  landing.  ROTC  students 
had  significantly  more  such  landings  than  any  other  group  except  T-38  graduates  (p-values  range  from 
<  0.01  to  <  0.05).  Private  pilots  also  had  significantly  fewer  nose  wheel  landings  than  T-38  graduates 
(p  =  0.038). 


Hard  Landings.  Figure  16  shows  the  proportion  of  hard  landings,  defined  as  landings  with  an 
instantaneous  sink  rate  of  480  ft/min  or  greater  at  the  moment  the  first  wheel  touched  the  runway.  The 
overall  effect  of  pilot  group  was  significant  (p  <  .02).  Pairwise  comparisons  between  the  groups  revealed 
that  T- 1  graduates  had  significantly  more  hard  landings  than  all  other  groups  except  the  Predator  pilots 
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(p  =  .076)  and  the  ROTC  students  (all  p’s  <  .01).  One  other  pairwise  comparison  approached,  but  did 
not  achieve,  significance  at  the  .05  level:  private  pilots  performed  fewer  hard  landings  than  the  ROTC 
students  (p  =  .063). 

Dangerous  Touchdown  Heading.  Landings  with  high  heading  deviation,  crab,  and/or  high 
lateral  velocity  could  result  in  excessive  gear  side-loading  and/or  potentially  running  off  the  side  of  the 
runway.  These  conditions  are  the  result  of  either  a  large  crab  angle  at  touchdown  and/or  a  velocity  vector 
substantially  misaligned  with  respect  to  the  runway  centerline.  Figure  17  shows  the  proportion  of 
landings  in  which  the  heading  deviation  at  touchdown  was  unacceptably  large  (greater  than  5  degrees). 
The  overall  effect  of  pilot  group  was  significant  (p  <  .01),  largely  due  to  the  ROTC  students’  overwhelm¬ 
ingly  poor  heading  deviation  performance  relative  to  all  the  other  groups  (all  p’s  <  .001).  No  other 
group  pairwise  comparisons  were  significant.  Lateral  velocity  under  wind  conditions,  shown  in  Figure 
18  (upper  line),  should  be  related  to  heading  deviations.  As  was  the  case  with  high  heading  deviation 
landings,  ROTC  students  performed  significantly  worse  (i.e.,  the  highest  average  lateral  velocity)  than  all 
other  groups  (all  p’s  <  .02),  while  all  other  group  pairwise  comparisons  were  not  significant. 
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Figure  15.  Proportion  of 
Nose-Wheel  First  Landings. 
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Figure  16.  Proportion  of  Flard 
Landings.  A  hard  landing  was 
defined  as  an  instantaneous 
sink  rate  at  the  moment  of 
touchdown  exceeding  480  feet 
per  minute. 
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Figure  17.  Proportion  of  High 
Heading  Deviation  Landings. 
A  high  heading  deviation  was 
defined  as  an  absolute  head¬ 
ing  at  touchdown  greater  than 
five  degrees  off  of  the  runway 
heading. 
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Proportion  Lateral  Velocity  (knots) 
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Figure  18.  Lateral 
Velocity  at  Touch¬ 
down.  Lateral 
Velocity  reported 
here  was  the  direct 
90-degree  velocity 
vector  to  the 
runway  heading. 
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Figure  19.  Pro¬ 
portion  of  Wing- 
Low  Landings. 

A  wing-low  land¬ 
ing  was  defined 
as  a  bank  angle 
greater  than 
seven  degrees  at 
the  moment  of 
touchdown. 
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Wing-Low  Landings.  Figure  19  shows  the  proportion  of  wing-low  landings,  in  which  the 
bank  at  touchdown  was  greater  than  seven  degrees.  Both  wing-low  and  high-lateral-velocity  landings 
occurred  predominately  on  trials  with  wind.  Though  the  overall  effect  of  pilot  group  regarding  wing-low 
landings  was  not  significant,  Predator  pilots  performed  very  few  wing-low  landings,  while  landings  with 
crosswind  for  all  other  groups  resulted  in  wing-low  landings  5%  or  more  of  the  time.  It  is  possible  that 
the  non-Predator-pilot  groups  had  more  difficulty  coordinating  the  interaction  of  bank,  rudder,  and  crab 
with  the  responsive  Predator  UAV. 

Landing  Phases.  To  further  explore  group  differences  in  landing  performance,  we  separated  the 
landing  into  phases — pattern,  final  approach,  roundout/flare,  and  touchdown.  Because  the  downwind 
and  base  legs  of  the  approach  pattern  are  quickly  learned  and  are  relatively  easy  to  fly,  results  show  no 
important  differences  between  groups  in  either  the  pattern  RMS  or  pattern  altitude  gate  measures;  that 
is,  a  ceiling  effect  was  found  where  all  groups  consistently  performed  at  or  better  than  criterion  level. 
Differences  between  the  groups  on  the  Landing  Task  became  apparent  during  the  more  difficult  stages 
(final  approach  through  touchdown),  especially  between  the  extreme  ends  of  the  experience  continuum 
(Predator  pilots  and  ROTC  students). 

Compared  to  other  groups  on  final  approach,  Predator  pilots  are  far  more  precise,  having  the 
smallest  groundtrack  and  glideslope  error,  maintaining  the  most  constant  airspeed  and  VSI  rates,  and 
having  significantly  more  stable  flight  characteristics,  that  is,  lower  variability  and  better  smoothness 
scores  (p-values  range  from  0.001  to  0.02).  Figures  20,  21,  22,  and  23  compare  the  groups  on  the  major 
final  approach  groundtrack/glideslope  measures.  Figures  24  and  25  show  variability  and  smoothness 
scores  for  the  final  approach  phase. 

Group  differences  in  performance  continue  to  develop  during  the  roundout/flare  phase.  In 
particular,  Predator  pilots  perform  the  roundout/flare  differently  than  other  groups.  They  do  a  more 
aggressive,  distinct  roundout/flare  by  waiting  longer  before  rounding  out  pitch,  but  flying  less  distance 
when  just  above  the  runway.  Figure  26  shows  the  large  difference  in  distance  flown  when  the  aircraft  is 
under  4-ft  above  ground  level  (AGL).  Predator  pilots  perform  the  distinct  roundout  and  flare  by  holding 
higher  airspeeds  and  greater  sink  rates  longer,  waiting  until  they  are  closer  to  the  runway,  yet  are  able  to 
bleed  off  this  speed  and  sink  rate  quickly  to  touch  down  the  softest.  This  is  shown  in  Figures  27,  28,  29, 
30,  and  31,  which  show  the  mean  vertical  sink  indicator  (VSI)  at  15-ft  AGL,  10  ft,  5  ft,  under  4  ft,  and  the 
instantaneous  sink  rate  at  touchdown,  respectively.  Finally,  as  was  the  case  for  final  approach,  Predator 
pilots  demonstrate  more  stable  flight  characteristics  during  roundout  and  flare.  The  same  variability  and 
smoothness  trends  as  those  found  for  the  glideslope  phase  (Figures  24  and  25)  were  found  for  the  landing 
attitude  phase  (under  4-ft  AGL),  but  unlike  for  glideslope,  the  groups  overall  did  not  differ  significantly 
in  variability  and  smoothness  while  in  a  landing  attitude  under  4-ft  AGL. 

At  touchdown,  predator  pilots  continue  to  show  better  performance.  They  have  the  lowest  sink 
rate,  pitch  angle,  and  lateral  velocity  of  any  of  the  groups.  By  contrast  the  ROTC  nonpilots  are  markedly 
worse  on  many  touchdown  parameters  than  the  other  groups.  In  particular,  they  touch  down  with  the 
highest  lateral  velocity  (Figure  18),  the  greatest  distance  from  centerline  (Figure  32),  and  the  highest 
heading  deviation  (Figure  33).  These  problems  arise  largely  during  landings  in  crosswind. 
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Glideslope  RMS  Error  (feet)  RMS  Error  (feet) 


Figure  20.  Final 
Approach  Groundtrack 
RMS  Error. 
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Figure  2 1 .  Glideslope  RMS 
Error. 
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Glideslope  VSI  Variance  Glideslope  Airspeed  Variance 
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Figure  22.  Glideslope 
Airspeed  Variance. 
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Figure  23.  Glideslope  VSI 
Variance. 
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Standard  Rate  Variance  Composite  (z)  Standard  Variance  Composite  (z) 
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Figure  24.  Glideslope 
Standard  Variance  Com¬ 
posite. 
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Figure  25.  Glideslope 
Standard  Rate  Variance 
Composite. 
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VSI(ft/min)  Distance  Flown  (feet) 
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Figure  26.  Distance  Flown 
Under  Four  Feet  Above 
Runway  (AGL). 
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Figure  27.  VSI  at  15 
Feet  AGL. 
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VSI  (ft/min)  VSI  (ft/min) 


Figure  28.  VSI  at  10 
Feet  AGL. 
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Figure  29.  VSI  at  5 
Feet  AGL. 
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Instantaneous  Sink  Rate  (ft/min)  VSI  (ft/min) 
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Figure  30.  VSI  Under  4 
Feet  AGL. 


Figure  31.  Instantaneous 
Sink  Rate  at  Touchdown. 
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Figure  32. 

Distance  Off  of 
Runway  Centerline 
at  Touchdown. 
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Figure  33.  Abso¬ 
lute  Heading  Devi¬ 
ation  From 
Runway  Centerline 
at  the  Moment  of 
Touchdown. 
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The  fact  that  Predator  pilots  are  superior  in  so  many  of  these  detailed  comparisons  at  different 
phases  of  the  Landing  Task  confirms  the  validity  of  the  Predator  aircraft  simulation  underlying  these 
tasks.  The  differences  observed  between  the  Predator  pilots  and  the  Predator  selectees — both  highly 
experienced  military  pilots  differing  primarily  in  Predator  experience — must  therefore  be  attributable  to 
the  Predator-specific  landing  skills  that  the  Landing  Task  demands. 


Reconnaissance  Task 

The  overall  result  for  the  Reconnaissance  Task  was  presented  in  Figure  6.  As  stated  earlier,  the 
effect  of  pilot  group  was  highly  significant.  Predator  pilots  obtained  the  most  time  on  target,  but  not 
significantly  more  than  selectees  and  T-38  graduates.  These  three  groups  obtained  significantly  more 
time  on  target  than  did  civilian  pilots  and  nonpilots.  Further  analysis  shows  some  of  the  factors  that 
underlie  the  strong  effect  of  pilot  group  on  this  task. 

Overall  time  on  target  in  the  Reconnaissance  Task  is  determined  by  two  things:  (a)  the  amount  of 
target  viewing  time  during  each  “pass”  through  the  target-viewable  area,  and  (2)  the  number  of  passes 
through  the  target-viewable  area  that  can  be  accomplished  during  each  10-minute  trial.  Although  the 
general  effect  of  pilot  group  was  evident  in  both  of  these  factors,  there  were  some  key  differences  in 
the  pattern  of  results. 

Time  per  pass.  Figure  34  shows  the  mean  amount  of  time  per  pass  through  the  target-viewable 
area.  The  overall  effect  of  pilot  group  was  highly  significant  (p  <  0.001)  and  the  pattern  is  generally 
similar  to  the  overall  time  on  target  results,  except  that  Predator  pilots  had  less  (but  not  significantly  less) 
time  per  pass  than  either  selectees  or  T-38  graduates. 
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Figure  34.  Time  on  target 
in  the  Reconnaissance  task 
per  pass  through  the  target 
viewing  area. 


29 


More  time  per  pass  can  be  obtained  in  a  number  of  ways.  The  data  suggest  that  two  primary 
methods  used  by  some  pilots  were  (a)  slowing  down  in  the  target-viewable  area;  and  (b)  climbing  higher 
so  that  the  target-viewable  area  is  larger.  The  data  show  that  Predator  pilots  were  no  more  likely  to 
avail  themselves  of  these  methods  than  were  selectees  and  T-38  graduates.  Figures  35  and  36  show 
mean  airspeed  and  altitude  while  the  aircraft  was  in  the  target-viewable  area.  Overall  effect  of  pilot 
group  was  significant  for  both  of  these  variables  (airspeed,  p  <  0.001;  altitude  p  =  0.026).  It  is  possible 
that  Predator  pilots  did  not  consider  large  changes  in  altitude  and  airspeed  in  the  target  area  to  be  a 
realistic  strategy,  one  that  they  would  use  during  a  real  mission.  So,  for  them,  time  per  pass  may  not 
be  a  valid  indicator  of  skill. 
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Figure  35.  Average  air¬ 
speed  while  in  the  target 
viewing  area  in  the  Recon¬ 
naissance  task. 


Number  of  passes.  The  reason  that  Predator  pilots  got  the  most  overall  time  on  target  despite 
having  less  time  per  pass  than  some  other  groups  is  that  they  achieved  more  passes  through  the  target 
viewable  area.  Figure  37  shows  the  data  on  number  of  passes.  The  effect  of  pilot  group  was  significant 
(p  =  0.016).  Predator  pilots  got  significantly  more  target  area  passes  than  any  of  the  civilian  groups. 
There  are  at  least  two  reasons  for  differences  in  number  of  passes.  First,  it  is  clear  that  maneuvering 
more  aggressively  when  outside  the  target-viewing  area  can  allow  the  pilot  to  more  quickly  return  to  the 
target  area.  Several  variables  were  indicative  of  aggressive  maneuvering  outside  the  target  area  (e.g., 
mean  absolute  bank  angle,  rudder  inputs).  Figure  38  shows  group  comparisons  on  one  such  variable, 
mean  yaw  rate  outside  the  target-viewable  area.  The  overall  effect  of  pilot  group  on  this  variable  was 
highly  significant  (p  =  0.004),  with  Predator  pilots  having  the  highest  mean  yaw  rate. 


A  second  reason  for  getting  more  passes  on  the  target  was  that  some  pilots  seemed  to  be 
particularly  good  at  figuring  out  where  the  target-viewable  area  would  be  and  staying  oriented  with 
respect  to  this  area.  This  ability  was  reflected  in  the  number  of  times  that  pilots  had  to  switch  out  of 
target  camera  mode  to  get  a  look  around  with  the  nose  camera.  Figure  39  shows  the  data  on  the  number 
of  target-to-nose  camera  switches  per  scenario.  Predator  pilots  switch  to  the  nose  camera  less  than  half 
as  many  times  as  any  other  group. 
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Number  of  Passes  on  Target  Average  Altitude  in  Target  Area  (ft) 
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Figure  36.  Average  altitude 
in  the  target  viewing  area  in 
the  Reconnaissance  task. 
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Figure  37.  Mean  number 
of  passes  through  the  target 
viewing  area  per  scenario  in 
the  Reconnaissance  task. 
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Nose  -Target  Cam  Switches  Per  Scenario  side  Target  Area  (deg./sec.) 
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Figure  38.  Average  yaw 
rate  per  scenario  outside 
of  the  target  area  in  the 
Reconnaissance  task. 
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Figure  39.  Number  of 
switches  between  nose 
camera  and  target  camera 
per  scenario  in  the 
Reconnaissance  task. 


32 


These  results  validate  the  initial  task  analysis.  Unlike  the  Basic  Maneuvering  and  Landing  Tasks, 
the  Reconnaissance  Task  is  not  primarily  a  test  of  the  ability  to  control  the  aircraft  within  precise  error 
tolerances.  Instead,  the  task  appears  to  require  the  ability  to  stay  oriented  without  a  constant  forward- 
looking  view,  to  keep  in  mind  the  relative  positions  of  the  aircraft,  target,  cloud  hole,  and  target-viewable 
area  are,  and  a  feel  for  how  to  best  position  the  aircraft  to  maximize  target  viewing  time  and  minimize 
the  time  required  to  get  back  to  some  part  of  the  target-viewable  area  after  leaving  it. 


Demographic  Data 

The  demographic  questionnaire  is  presented  in  Appendix  C.  The  overall  result  of  analysis 
of  participants’  responses  is  that  the  demographic  characteristics  of  the  participants  generally  do  not 
relate  strongly  to  performance  on  our  simulated  Predator  tasks.  Significant  relationships  that  do  exist 
(e.g.,  between  age  and  performance)  are  mediated  by  pilot  group  membership  (e.g.,  ROTC  students  are 
younger  and  have  no  flight  experience). 

Part  of  the  questionnaire  concerned  hobbies  that  have  been  informally  mentioned  as  possible 
indicators  of  UAV  pilot  skill,  namely  frequency  and  type  of  videogame  playing,  and  experience  with 
remote-controlled  hobby  aircraft.  Experience  with  nonflight-related,  action-oriented  computer  games 
was  not  associated  with  better  performance  on  our  tasks.  Neither  was  experience  with  remote-controlled 
toy  aircraft.  However,  there  was  a  small  but  statistically  significant  relationship  between  number  of 
lifetime  hours  playing  flight  simulation  computer  games  and  landing  performance.  Those  with  more  self- 
reported  experience  playing  flight-related  computer  games  needed  fewer  trials  to  pass  the  Landing  Task 
criteria  (r  =  -.233,  p  =  0.022).  This  relationship  is  not  an  artifact  of  age  difference  across  groups.  In 
fact,  the  relationship  is  strongest  within  particular  groups,  especially  the  T-38  and  T-l  groups  (T-38:  r  = 
-.605,  p  =  0.017;  T-l:  r  =  -.582,  p  =  0.018;  Combined  T-38  &  T-l:  r  =  -.545,  p  =  0.002). 

These  results  do  not  prove  that  assigning  randomly  selected  pilots  to  play  computer  flight  games 
would  necessarily  make  them  better  at  flying  the  Predator.  Other  factors  such  as  motivation  to  practice 
flying,  or  to  excel  at  any  flying-related  challenge,  could  be  mediating  this  relationship. 


DISCUSSION 

Reference  Groups.  It  is  clear  that  the  tasks  chosen  and  the  underlying  simulation  were  of 
sufficient  fidelity,  sensitivity,  and  validity  to  produce  credible  assessments  of  the  ability  to  leam  to  fly  the 
Predator.  The  Predator  pilot  reference  group  performed  consistently  better  than  all  other  groups  in  both 
overall  measures  of  merit  and  detailed  measures  of  technique.  Conversely,  the  ROTC  nonpilot  reference 
group  performed  consistently  worse,  as  expected.  This  latter  result  provides  empirical  support  for  the 
conclusions  of  the  Hall  and  Tirre  (1998)  survey  of  Predator  pilots  regarding  the  necessity  of  some  prior 
flying  experience  in  piloting  UAVs. 

Candidate  Groups.  The  groups  of  primary  interest  in  this  study  were  those  that  would  be  likely 
alternative  candidates  for  selection  to  fly  the  Predator:  current  selectees,  T-38  and  T-l  graduates,  civilian 
instrument  pilots,  and  private  pilots.  On  the  major  performance  measures,  especially  the  number  of  trials 
required  to  pass  Basic  Maneuvering  and  Landing  Tasks,  these  groups  did  not  differ  dramatically.  In 
particular,  current  selectees,  T-38  graduates,  and  civilian  instrument  pilots  showed  nearly  equal  levels 
of  performance  on  many  measures.  The  high-level  similarity  of  the  performance  of  these  groups  was  not 
due  to  experimental  or  measurement  factors.  For  example,  tasks  were  at  the  right  level  of  difficulty  to 
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avoid  ceiling  or  floor  effects.  Differences  between  individuals  within  the  groups  were  high,  but  enough 
data  were  gathered  to  allow  the  average  performance  of  the  different  groups  to  be  established  with 
reasonable  precision.  Moreover,  the  pattern  of  overall  group  comparisons  was  replicated  in  many  fine¬ 
grained  supporting  measures  across  different  task  segments.  Therefore,  the  similarity  in  performance 
of  several  of  the  candidate  groups  is  a  solid  and  reliable  finding.  However,  there  is  one  major  caveat. 
Even  though  the  candidate  groups  did  not  differ  much  on  many  measures,  some  tasks  and  measures 
showed  differences  that  are  potentially  important.  We  now  discuss  relevant  specific  considerations  for 
each  of  the  non-Predator  pilot  groups. 


Predator  Selectees 

Perhaps  the  most  striking  finding  from  the  Predator  Selectee  group  is  that  they  did  not  perform 
significantly  better  than  some  of  the  other,  less  experienced  groups  (e.g.,  T-38  graduates,  civilian 
instrument  pilots)  on  most  measures.  This  reflects  the  fact  that  most  of  our  tasks  and  performance 
measures  tap  Predator-specific  handling  skills.  By  the  time  a  pilot  has  150-200  flight  hours,  he  or  she 
has  probably  already  developed  most  of  the  skills  that  are  needed  to  prepare  for  learning  basic  maneuvers 
and  landing  in  the  Predator.  Hundreds  more  hours,  especially  in  non-Predator  aircraft,  may  not  give  a 
pilot  a  substantial  extra  boost  on  basic  handling  tasks. 

Another  consideration  in  understanding  the  performance  of  this  group  is  that  the  selectees  are  not 
necessarily  a  representative  sample  of  all  Air  Force  pilots  with  at  least  one  operational  weapons  system 
tour.  Moreover,  while  some  of  the  selectees  may  have  volunteered  for  Predator  duty,  others  may  not  have 
been  volunteers,  which  could  affect  morale  and  motivation  not  only  for  their  upcoming  assignment,  but 
also  for  their  participation  in  our  tasks. 

Finally,  we  reiterate  that  our  study  does  not  measure  some  less  tangible  but  valuable  characteris¬ 
tics  of  pilots.  Officership,  airmanship,  communication  skills,  command  experience,  knowledge  of  combat 
operations,  familiarity  with  airspace  management,  especially  in  war  zones,  and  other  skills  gained  from 
one  or  more  operational  tours  may  weigh  in  favor  of  using  experienced  pilots  even  if  less  experienced 
groups  have  the  requisite  basic  piloting  skills.  In  this  regard,  since  the  navigators  currently  selected 
for  Predator  have  operational  experience,  it  is  unfortunate  that  a  sufficient  number  of  them  were  not 
available  to  form  a  separate  comparison  group. 

Since  selectees  did  not  perform  worse  than  other  candidate  groups,  the  Predator  flying  skill 
performance  data  reported  here  supports  current  selection  practices  and  provides  no  compelling  perfor¬ 
mance-based  reason  to  change  the  current  selection  policy.  Of  course  policy  changes  could  easily  be 
driven  by  cost  or  pilot  availability  issues,  hence  the  inclusion  of  other  groups  in  the  study. 


T-38  Graduates 

T-38  graduates  performed  quite  well  on  most  of  our  measures.  One  explanation  for  their 
success  at  Basic  Maneuvering  and  handing  is  that,  though  there  are  certainly  many  differences,  the 
T-38  aircraft  has  some  important  similarities  to  the  Predator  in  handling  characteristics  and  technique. 
Control  by  stick,  sensitivity  of  the  stick  (i.e.,  responsive  bank/pitch  reaction  by  the  aircraft),  aggressive 
roundout/flare,  shorter  aimpoint  at  landing,  and  transition  to  landing  with  idle  power  are  characteristic 
of  both  aircraft. 


34 


T-38  graduates  also  performed  well  on  the  Reconnaissance  Task,  which  was  presented  after  Basic 
Maneuvering  and  Landing  training,  and  should  be  less  sensitive  to  aircraft  handling  issues.  Analysis 
of  subsidiary  performance  measures  on  this  task  showed  that  T-38  pilots  obtained  extra  time  on  target 
by  maneuvering  more  aggressively  than  the  civilian  instrument  and  private  pilot  groups.  They  climbed 
to  higher  altitudes  in  the  target  area,  cut  airspeed  more  drastically  when  necessary,  and  made  more 
aggressive  turns  outside  the  target  area  to  return  as  quickly  as  possible. 

T-38  pilots  would  be  an  acceptable  candidate  group  for  selection  as  Predator  pilots  if  there  were 
a  way  for  them  to  obtain  critical  knowledge  about  combat  operations  equivalent  to  that  achieved  by  a 
prior  operational  tour.  The  responsiveness  of  the  T-38  to  control  surface  manipulations  appears  to  allow 
considerable  transfer  of  skills  to  the  Predator. 


T-l  Graduates 

On  many  of  our  measures,  T-l  graduates  performed  less  well  than  other  groups  with  similar 
number  of  flight  hours.  Analysis  of  the  detailed  performance  measures  suggest  that  experience  with  the 
T-l  aircraft  does  not  transfer  as  well  to  the  Predator  as  does  experience  with  T-38  or  propeller  aircraft. 
Unlike  the  T-38  and  Cessna  172,  the  T-l  appears  to  have  few  readily  identifiable  similarities  to  the 
Predator.  As  a  larger,  yoke-controlled  aircraft,  the  T- 1  has  more  weight  and  inertia,  thereby  tending  to 
dampen  and  delay  the  effect  of  control  surface  manipulation.  Also  the  transition  to  land  is  more  subdued 
than  the  aggressive  roundout/flare  performed  by  Predator  pilots,  T-38,  and  civilian  Cessna  172  pilots. 
Another  indication  that  the  performance  of  T-l  pilots  was  affected  by  differences  in  aircraft  handling  is 
that,  on  the  Reconnaissance  Task,  which  is  less  dependent  on  precise  aircraft  handling,  the  T- 1  graduates 
outperformed  the  civilian  groups.  (Another  possible  explanation  for  the  slightly  worse  performance  by 
the  T-l  pilots  relative  to  T-38  pilots  on  most  of  our  tasks  and  measures  is  that  the  majority  of  the  top 
ranked  students  from  T-37  classes  may  choose  and  receive  assignment  to  T-38  instead  of  T-l). 

If  prior  operational  experience  were  not  required,  T- 1  graduates  would  be  an  acceptable  choice 
from  among  the  candidate  groups  for  selection  to  fly  Predator.  However,  based  upon  the  performance 
data  observed  in  this  study,  they  would  probably  not  be  the  first  choice,  possibly  due  to  one  of  the 
speculative  factors  mentioned  above.  Nevertheless,  since  many  successful  Predator  pilots  originated 
from  T-ls  and  other  aircraft  with  handling  characteristics  different  than  the  Predator,  it  is  clear  that  these 
handling  differences  can  be  overcome. 


Civilian  Instrument  Pilots 

Civilian  instrument  pilots  have  almost  the  same  amount  of  flying  experience  as  T-38  and  T-l 
graduates,  but  of  a  different  type.  Although  they  have  slightly  fewer  average  number  of  flying  hours 
than  UPT  graduates,  they  perform  more  actual  aircraft  landings  during  training  (about  240  for  civilians 
versus  about  210  for  UPT  graduates).  Though  there  are  obviously  many  differences  between  civilian 
propeller  planes  and  the  Predator,  there  are  some  identifiable  factors  that  may  positively  transfer  to 
flying  the  Predator,  such  as  relatively  lightweight,  propeller-driven  dynamics,  and  aggressive  transition 
to  land  with  a  short  aimpoint. 

This  group,  like  the  T-38  pilots,  would  be  a  suitable  choice  for  Predator  training  if  operational 
experience  were  not  required.  The  flying  skill  demonstrated  by  the  civilian  instrument  group  would 
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support  further  investigation  of  a  number  of  alternative  Predator  selection  groups,  including,  but  not 
limited  to,  new  UAV-specific  career/training  paths  for  other  non-UPT  graduate  Air  Force  officers,  or 
the  use  of  contractors.  Since  navigators  currently  selected  for  Predator  pilot  duty  have  at  least  as 
much  light  aircraft  flight  experience  as  the  civilian  instrument  group  in  this  study,  the  results  from  the 
civilian  instrument  group  may  also  provide  some  support  for  the  current  practice  of  selecting  navigators. 
However  an  important  distinction  is  that  all  the  pilots  in  the  civilian  instrument  group  had  just  completed 
their  training,  whereas  navigators  may  have  as  much  or  more  total  flight  experience,  but  less  recent 
experience. 


Civilian  Private  Pilots 

Private  pilots  do  fairly  well  on  some  of  our  landing  measures,  perhaps  because  private  pilot 
training  emphasizes  repeated  landings.  They  were  the  only  group  in  our  study  with  such  frequent  recent 
landing  emphasis.  However  the  Landing  Task  also  demanded  some  instrument  reference  and  calculation 
skills  that  presumably  are  developed  to  a  greater  extent  in  pilots  with  instrument  ratings.  This  deficit  also 
could  underlie  the  private  pilots’  relatively  poor  performance  on  the  Basic  Maneuvering  Task. 

Private  pilots  would  be  the  least  suitable  of  any  group  we  tested  (except  the  nonpilot  ROTC 
group)  for  selection  as  Predator  pilots.  Their  lack  of  instrument  flight  experience  is  apparent  in 
performance  on  our  tasks. 


ROTC  Students 

As  mentioned  previously,  learning  to  fly  a  UAV  may  be  problematic  for  some  pilots  because 
they  cannot  rely  on  some  cues  that  are  normally  available  in  manned  aircraft  (e.g.,  vestibular  cues,  field 
of  view,  etc.).  In  this  sense,  the  ROTC  students  may  actually  have  had  some  limited  advantage  over 
pilots  when  first  introduced  to  flying  the  Predator  UAV  simulation.  However,  the  disadvantage  of  lack 
of  flying  experience  outweighed  any  potential  advantage.  In  all  of  the  tasks,  ROTC  students  performed 
worse  than  any  other  group.  Despite  declarative  and  procedural  training  and  supplemental  instructional 
documents,  several  participants  in  this  group  were  unable  to  fully  understand  the  complex  relationship 
between  bank,  rudder,  and  crab  angle  in  crosswind  landing,  some  even  after  200  landing  trials  and 
repeated  voluntary  reviewing  of  the  crosswind  instructional  video.  Also,  many  ROTC  participants 
were  not  fully  aware  of  the  consequences  of  last-minute  corrections  on  landings.  They  seemed  to 
underestimate  the  magnitude  and  widespread  ramifications  of  the  corrections  they  made.  Nevertheless, 
all  ROTC  participants  were  able  to  pass  the  crosswind  Landing  Task  eventually.  This  often  seemed 
to  be  the  result  of  a  very  scripted  effort  to  pass  one  step  at  a  time  within  a  landing  trial,  rather  than 
understanding  the  complex  interactions  necessary  to  recover  from  any  situation.  Situations  in  which  the 
aircraft’s  attitude  or  position  was  very  far  off  of  the  “current  step”  often  lead  to  overcontrol,  frequently 
resulting  in  disastrous  consequences.  Therefore,  they  may  have  developed  strategies  for  passing  the  task 
that  would  not  generalize  to  comprehensive  skill  with  a  real  aircraft. 

The  ROTC  group  was  included  in  this  study  for  two  specific  purposes.  First  the  ROTC  students 
are  a  referent  nonpilot  group  to  verify  that  the  tasks  reflect  piloting  skill.  The  data  support  this  assertion, 
since  ROTC  students  performed  worse  than  pilots  on  both  overall  performance  measures  and  detailed 
indications  of  aircraft  control  skill.  These  differences  were  found  even  though  all  groups  were  provided 
with  declarative  and  procedural  training. 
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Secondly,  the  performance  of  the  ROTC  group  provides  some  relative  indication  of  how  long  it 
might  take  to  train  a  nonpilot  up  to  a  criterion  skill  proficiency  level.  Not  surprisingly,  based  upon  the 
flying  skill  demonstrated  in  this  study,  ROTC  students  are  not  a  suitable  candidate  group  for  Predator 
UAV  selection  unless  a  specialized,  lengthy  Predator  flight  training  program  were  to  be  initiated.  They 
did,  however,  perform  somewhat  better  than  anticipated,  especially  on  the  Basic  Maneuvering  and  Land¬ 
ing  Tasks.  Pilots  from  the  T-38,  T-l,  and  civilian  instrument  programs  normally  receive  200+  landings 
before  graduation,  yet  the  ROTC  students  were  able  to  pass  the  Basic  Maneuvering  and  Landing  Tasks 
in  about  161  attempts.  This  was  probably  not  due  to  the  task  being  easy;  several  of  the  experienced 
military  pilots  in  this  study  reported  the  Landing  Task  as  “the  most  difficult  piloting  task  I’ve  ever  had  to 
do.”  Rather,  it  may  be  that,  while  many  aspects  of  pilot  skill  transfer  to  flying  the  Predator,  other  aspects 
may  not.  Experienced  pilots  may  need  to  “unlearn”  some  aspects  of  piloting  (e.g.,  use  of  vestibular  and 
peripheral  cues),  whereas  nonpilots  who  train  only  on  the  Predator  would  not. 


CONCLUSION  AND  GENERAL  RECOMMENDATIONS 

The  results  of  this  study  suggest  that  150-200  hours  of  recent  previous  flying  experience  is  suf¬ 
ficient,  on  average,  to  prepare  a  pilot  to  learn  to  fly  a  high-fidelity  Predator  simulation— if  this  experience 
is  obtained  in  an  aircraft  with  handling  characteristics  similar  to  those  of  the  Predator.  In  particular, 

T-38  graduates  and  civilian  instrument  pilots  performed  nearly  as  well  as  current  Predator  selectees  on 
difficult  aircraft  handling  tasks  in  the  simulator.  This  finding  was  to  some  degree  unexpected,  and  it 
has  possible  implications  for  Air  Force  selection  policy.  Some  general  recommendations  based  on  the 
results  of  the  study  are: 

1 .  If  prior  operational  experience  remains  a  prerequisite  for  assignment  to  Predator,  then  the  current 
practice  of  selecting  experienced  pilots  should  be  continued.  Current  selectees  performed  at  least  as  well 
as  any  other  candidate  group  in  this  study,  and  better  than  some. 

2.  If  critical  knowledge  about  combat  operations  equivalent  to  a  prior  operational  tour  could  be  provided 
to  candidate  pilots,  then,  from  a  pure  aircraft  control  skills  perspective,  either  T-38  graduates  or  civilian 
instrument-trained  pilots  should  be  considered  as  a  potential  source  of  Predator  pilots.  However,  in 
either  instance,  we  would  recommend  that  a  test  sample  of  at  least  15  such  pilots  be  trained  in  the 
Predator  IQT  and  MQT  programs,  and  the  results  evaluated,  before  any  official  change  in  policy  is 
made. 

3.  Navigators  with  instrument  ratings  may  be  an  especially  attractive  group  to  select  from  due  to  their 
operational  experience,  but  we  had  insufficient  data  from  them.  We  recommend  that  an  additional  1 1 
navigator  selectees  undergo  our  study  protocol  so  that  a  complete  picture  of  the  capabilities  of  current 
selectees  can  be  obtained.  At  present,  our  conclusions  about  the  skill  of  current  selectees  only  apply  to 
those  selectees  with  prior  operational  assignments  as  pilots. 

4.  We  recommend  that  a  cognitive  task  analysis  of  Predator  combat  operations  be  conducted  with  the 
goal  of  specifying  the  operational  knowledge  and  skills  that  a  Predator  pilot  must  obtain,  either  from  a 
prior  tour  or  an  apprenticeship  program.  This  would  include  analysis  of  command  and  communications 
knowledge  and  skills,  airspace  management  in  war  zones,  and  other  specific  knowledge  about  the 
conduct  of  combat  operations. 
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APPENDIX  A:  DESCRIPTION  OF  SYNTHETIC  TASKS 


The  purpose  of  this  section  is  to  provide  a  description  of  some  realistic  unmanned/uninhabited 
aerial  vehicle  (UAV)  pilot  synthetic  tasks  developed  by  the  Warfighter  Training  research  Division  of 
the  Air  Force  Research  Laboratory  (AFRL/HEA).  We  recommend  these  tasks  to  researchers  who 
wish  to  contribute  to  a  body  of  knowledge  about  human  performance  issues  in  piloting  UAVs.  These 
tasks  were  built  upon  a  detailed  simulation  of  the  Predator  UAV  developed  by  AFRL-Mesa  via  its 
on-site  contractors  and  Parker  International.  This  simulation  is  now  being  used  to  train  all  Air  Force 
Predator  pilots. 

The  design  of  these  synthetic  tasks  results  from  a  unique  collaboration  between  experimental 
psychologists  and  expert  pilots  of  the  Air  Force’s  Predator  UAV.  Tests  using  military  and  civilian  pilots 
showed  that  experienced  UAV  pilots  perform  better  on  these  tasks  than  pilots  highly  experienced  in  other 
aircraft,  indicating  that  the  tasks  tap  UAV-specific  pilot  skill. 

Our  aim  in  developing  these  synthetic  tasks  was  to  extract  important  segments  of  the  UAV  pilot’s 
overall  task — segments  that  tax  the  key  skills  required  by  a  UAV  pilot.  We  wanted  to  provide  tasks 
that  lend  themselves  to  laboratory  study,  yet  do  not  fall  prey  to  oversimplifications  that  change  the 
fundamental  skills  and  strategies  required  to  fly  a  real  UAV.  Our  design  philosophy  and  techniques  are 
described  in  Martin,  Lyon,  and  Schreiber  (1998). 

All  of  our  UAV  synthetic  tasks  use  the  same  equipment.  The  equipment  specifications  are  shown 
in  Table  1.  Figure  1  shows  the  experimental  setup. 

Table  1 :  Equipment  List 

Participant  Station  (runs  synthetic  tasks)  Dual  333  MHz  Pentium  II  w/  5 12K  cache 

440  LX  Motherboard  P6DLS 

256  MB  ECC  SDRAM 

24x  EIDE  CD-ROM 

300  Watt  Power  Supply 

SoundBlaster  16  Sound  card 

10/100  lOBaseT  Ethernet  card 

1 .44  MB  Floppy  drive 

Windows  NT  Workstation  4.0 

NT  Service  Pack  5 

Internal  100  MB  Zip  drive 

8.2  GB  Maxtor  EIDE  disk  Drive 

Two  Elsa  Gloria  XL  Graphic  cards 

Two  16  MB  Memory  Upgrade  for  ELSA  Gloria  XL 

1  Additional  fan  kit 

Researcher  Station  (tutorials,  data  storage)  Dual  200  MHz  Pentium  Processors 

128  MB  ECC  SDRAM 
SoundBlaster  16  Sound  card 
10/100  lOBaseT  Ethernet  card 
1 .44  MB  Floppy  drive 
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Windows  NT  Workstation  4.0 
NT  Service  Pack  5 
Standard  Graphics  card 
CD-ROM 

8.2  GB  Maxtor  EIDE  disk  Drive 
1 .44  MB  Floppy  drive 
Internal  100  MB  Zip  drive 
1  Additional  fan  kit 

Additional  Hardware  Two  25’  lOBaseT  Ethernet  Cables 

One  100  MHz  Ethernet  Hub 
Three  1 10V  AC  Surge 

Three  17”  CRT  Monitors  (CTX  VL-710,  0.25mm) 

Joystick  (Thrustmaster  PFCS  serial) 

Rudder  (Thrustmaster  Elite  Rudder  Pedals) 

Throttle  (Thrustmaster  WCS) 

The  apparatus  consists  of  a  flight  control  stick  at  the  right  hand,  a  throttle  at  the  left  hand, 
rudders  (not  shown),  a  keyboard,  and  two  adjacent  19”  color  monitors.  During  each  task,  the  left-hand 
monitor  displays  the  Predator  UAV  primary  flight  control  instrumentation,  sometimes  superimposed  over 
simulated  video  from  the  sensor  camera.  The  right-hand  monitor  often  contains  a  map  display  but  can 
show  other  information  depending  on  the  task.  (The  computer  on  the  right  is  the  experiment  control 
station,  which  is  normally  outside  the  pilot’s  view  on  another  table.) 

All  of  the  tasks  have  a  common  general  structure.  The  pilot  first  works  through  a  computer-based 
multimedia  tutorial  that  provides  information  necessary  to  interpret  the  displays  and  perform  the  tasks. 
When  he  or  she  is  ready  to  begin  the  task,  an  initial  display  screen  shows  information  about  the  trial 
that  is  about  to  begin.  When  the  pilot  is  ready,  he  or  she  presses  the  trigger  on  the  control  stick  and 
the  trial  begins.  After  the  trial  finishes,  a  feedback  screen  is  presented  showing  the  results  of  detailed 
performance  measures. 

We  will  describe  three  synthetic  tasks — the  Basic  Maneuvering  Task  in  which  a  pilot  must  make 
controlled  changes  in  UAV  airspeed,  altitude,  and  heading;  the  Landing  Task  in  which  the  UAV  must  be 
guided  through  a  standard  approach  and  landing;  and  the  Reconnaissance  Task  in  which  the  goal  is  to 
obtain  simulated  video  of  a  target  through  a  small  break  in  cloud  cover. 


Basic  Maneuvering  Task 

The  Basic  Maneuvering  Task  was  designed  to  train  people  with  various  types  and  amounts  of 
flying  experience  up  to  a  standard  level  of  basic  Predator  UAV  flying  proficiency.  After  achieving  this 
proficiency  standard,  more  specific  Predator  UAV  piloting  skills  can  be  tested  with  other  synthetic  tasks. 
The  structure  of  the  Basic  Maneuvering  Task  was  adapted  from  a  task  designed  at  the  University  of 
Illinois  to  examine  scan  patterns  of  pilots  who  differed  in  experience  flying  a  small  aircraft  (Wickens, 
Bellenkes,  &  Kramer,  1995).  However  the  current  task  uses  Predator  UAV  instrument  displays  and 
includes  extensive  performance  measurement  and  feedback  information. 


The  Basic  Maneuvering  Task  is  a  set  of  timed  basic  flying  maneuvers.  The  task  requires  the  participant 
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Figure  1.  Predator 
Synthetic  Task 
Environment  equip¬ 
ment  setup. 


to  fly  seven  distinct  maneuvers  while  maintaining  acceptable  root-mean-squar  (RMS)  error  limits  on  the 
parameters  of  altitude,  airspeed,  and  heading.  Each  maneuver  starts  with  a  10-second  straight-and-level, 
lead-in  section  as  the  participant  prepares  to  execute  the  maneuver.  At  the  end  of  this  lead-in,  the  timed 
maneuver  segment  (either  60  or  90  seconds)  begins  and  the  participant  is  required  to  maneuver  the 
aircraft  at  a  constant  rate  of  change  with  regard  to  each  of  the  three  parameters.  The  initial  segments 
require  the  participant  to  perform  maneuvers  that  change  in  one  parameter  while  holding  the  other  two 
constant.  For  example,  the  participant  may  be  required  to  reduce  airspeed  while  maintaining  altitude 
and  heading.  The  maneuvers  increase  in  complexity  by  requiring  the  participant  to  fly  maneuvers  that 
change  in  combinations  of  two  parameters.  The  final  maneuver  requires  changing  all  three  parameters 
simultaneously.  The  participant  is  not  allowed  to  continue  on  to  the  next  maneuver  until  the  RMS 
criterion  error  limits  have  been  achieved  on  all  three  parameters  for  the  current  maneuver.  These  limits 
were  chosen  after  the  tasks  were  flown  by  highly  experienced  Predator  pilots.  The  idea  was  to  set 
performance  goals  that  could  be  achieved  by  a  skilled  Predator  pilot  after  a  few  familiarization  trials. 
These  limits  are  easy  for  a  programmer  to  modify  if  desired. 

Figures  2  and  3  show  the  displays  that  appear  while  someone  is  performing  the  Basic  Maneuvering  Task. 
The  left  monitor  depicts  the  Predator  flight  information  overlaid  on  a  pure  black  background,  requiring 
the  participant  to  fly  the  tasks  by  instruments  only  (Figure  2).  The  pertinent  flight  information  is  altitude, 
digital  heading,  vertical  speed  indicator,  artificial  horizon,  heading  rate  indicator,  and  engine  revolutions 
per  minute.  The  right  monitor  displays  other  useful  information,  such  as  a  moving  compass,  a  bank 
angle  indicator,  the  time  remaining  in  the  maneuver,  and  the  start/end  parameter  values  for  the  maneuver 
currently  being  flown  (Figure  3).  At  the  end  of  the  maneuver,  the  right  monitor  displays  feedback  to 
the  participant  (Figure  4). 

Both  the  optimal  and  the  actual  parameter  values  over  time  are  graphed  for  each  of  the  three  parameters, 
with  time  on  the  abscissa  and  the  value  of  the  parameter  on  the  ordinate.  The  average  RMS  error  for 
each  parameter  is  also  shown.  If  a  participant  failed  any  parameter  on  that  maneuver,  the  corresponding 
RMS  error  value  will  flash  on  and  off.  Once  the  participant  passes  all  seven  Basic  Maneuvering  Task 
segments,  he/she  moves  on  to  the  Landing  Tutorial  and  the  Landing  Task. 
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Figure  2.  Predator 
“Heads-Up  Display” 
overlaid  on  black 
background. 


TRIAL  INITIATED 

HOLD  ALTITUDE  Ar  15000  FEET 
CHANGE  AIRSPEED  FROM  67  TO  62  KIAS 
HOLD  HEADING  AT  0  DEGREES 


'1 

CUKKI-NI  LEG 
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1  ...0!. 
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1 _ 1 

Figure  3.  An  example  of 
what  is  displayed  on  the 
right-hand  monitor  during 
the  Basic  Maneuvering  Task. 
In  this  particular  instance, 
the  maneuver  is  to  reduce 
airspeed  while  holding  alti¬ 
tude  and  heading  constant. 
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Current  Leg:  3  j 
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Figure  4.  An  example 
of  the  feedback  for  the 
Basic  Maneuvering  Task. 
Criterion  performance  levels 
shown  in  brackets,  while 
the  participant’s 
performance  is  graphed  and 
displayed  in  yellow.  In 
this  particular  example  the 
maneuver  asked  for  increas¬ 
ing  altitude  only  while  hold¬ 
ing  airspeed  and  heading 
constant. 


Landing  Task 

Perhaps  the  most  difficult  task  associated  with  the  Predator  UAV  is  landing.  In  addition  to  flying 
with  a  line-of-sight  control  delay,  remotely  controlling  the  aircraft  on  landings  leaves  the  pilot  with 
no  vestibular  cues  to  feel  the  flare  and  sinking  feelings,  and  no  peripheral  visual  cues  to  notice  the 
surrounding  terrain  rising  in  the  last  few  moments  before  touchdown.  These  factors  complicate  the 
pilot’s  task  of  controlling  a  slow-moving  aircraft  with  a  long  wingspan  that  is  naturally  susceptible  to 
crosswinds  and  other  disturbances.  Our  Landing  Task — our  most  difficult  synthetic  task — replicates 
most  of  the  difficulties  encountered  by  an  operator  of  the  actual  Predator  UAV  during  an  approach  and 
landing  (except  that  turbulence  and  ground  effects  are  not  simulated).  It  requires  the  pilot  to  exceed 
criterion  level  of  performance  on  13  different  parameters  on  the  same  landing  attempt,  first  for  a  no-wind 
landing  and  then  a  cross-wind  landing. 

The  participant  is  asked  to  fly  the  approach  pattern,  maintain  glideslope,  and  safely  land  the  aircraft. 

At  the  start  of  each  landing  attempt,  the  participant’s  plane  is  initialized  at  800  feet  with  the  gear  down 
on  the  downwind  leg  of  one  of  the  landing  patterns  Predator  pilots  used  at  Indian  Springs  Air  Force 
Auxiliary  Field  (AFAF)  (Figure  5).  The  Predator  flight  displays  are  overlaid  on  the  simulated  30-degree 
nose  camera  imagery  and  displayed  to  the  participant  on  the  left  monitor  (Figure  6). 

The  13  parameters  that  must  be  met  are:  RMS  pattern  groundtrack  error,  altitude  at  three  different 
points  in  the  pattern,  RMS  final  approach  groundtrack  error,  RMS  final  approach  glideslope  error, 
touchdown  bank  angle,  touchdown  pitch  angle,  touchdown  groundspeed,  touchdown  instantaneous  sink 
rate,  touchdown  heading  relative  to  runway,  touchdown  distance  off  centerline,  and  lateral  movement 
at  touchdown  (measure  of  side-loading).  Feedback  on  all  these  parameters  is  shown  on  the  right-hand 
monitor  at  the  conclusion  of  the  landing  attempt  (Figure  7). 

As  was  the  case  for  the  Basic  Maneuvering  Task,  if  the  participant  fails  one  or  more  parameters,  the 
task  is  flown  again  until  criterion  performance  is  attained  on  all  parameters  on  the  same  landing  attempt. 
Once  the  participant  has  passed  a  no-wind  landing  attempt,  then  passes  one  of  the  cross-wind  landing 
conditions  (of  four),  he/she  moves  on  to  the  Reconnaissance  Tutorial  and  the  Reconnaissance  Task. 
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Figure  5.  The  tracker  map 
is  displayed  on  the  right-hand 
monitor.  This  example 
shows  a  plane  on  final 
approach.  The  participant 
sees  only  the  map  and  plane 
icon;  we  have  superimposed 
the  approach  path  (red  line), 
turn  points  (red  dots)  and 
altitude  gate  positions  (black 
lines)  for  illustrative  pur¬ 
poses  only. 


Figure  6.  The  forward- 
looking  nose  camera 
view  with  Predator  dis¬ 
plays.  Shown  here  is 
the  aircraft  on  final 
approach. 
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Heading : 

[89.6  -  91.6] 
Groundspeed : 
[70.0  Knots] 
Bank  Angle: 
[1-3] 


Vertical  Velocity: 

[fps  -3.7  (fpm  -220.0)] 
Distance  off  center: 
[10.0  Feet] 

Lateral  Movement: 

[-0.80  to  0.80  Knots] 


P i tch : 


[1.0  to  4.0] 


Figure  7.  Feedback  for 
each  of  the  13  Landing 
Task  parameters.  Criterion 
performance  levels  shown 
in  brackets,  while  the  par¬ 
ticipant’s  performance  is 
graphed  and  displayed  in 
yellow. 


Reconnaissance  Task 


One  frequent  and  difficult  aspect  of  an  operational  reconnaissance  mission  is  getting  electro-optical  (EO) 
imagery  of  a  target  through  holes  in  clouds.  The  skill  of  the  pilot  is  particularly  important  in  this 
situation,  since  he/she  must  maneuver  the  UAV  such  that  the  UAV,  the  target,  and  a  cloud  break  are 
all  aligned  (Figure  8)  in  order  to  obtain  time  on  target  (TOT)  with  the  EO  camera.  A  few  factors 
adding  to  the  difficulty  of  this  task  are  the  unpredictable  location  of  the  cloud  break,  the  effects  of  wind 
on  the  UAV,  no-fly  zones,  altitude  and  time  restrictions,  threat  rings,  terrain,  and  the  maneuverability 
constraints  of  the  Predator  itself. 


Figure  8.  Schematic  drawing 
of  the  Reconnaissance  Task. 
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In  our  synthetic  Reconnaissance  Task,  the  participant  encounters  these  same  difficulties.  The  Reconnais¬ 
sance  Task  consists  of  10-minute  missions  (trials)  in  which  the  participant  must  maximize  time  on  target 
(TOT)  while  taking  into  account  various  constraints  and  hazards  in  his/her  attempt  to  align  the  Predator, 
the  cloud-break,  and  the  target.  Time  spent  in  violation  of  altitude,  airspeed,  or  no-fly  zones  results 
in  penalty  time,  which  is  subtracted  from  accumulated  TOT.  In  addition,  the  participant  is  instructed 
to  avoid  situations  that  would  ordinarily  make  it  difficult  (in  the  real  Predator  team  setting)  for  the 
camera  operator  to  obtain  quality  EO  imagery.  Therefore  TOT  is  not  accumulated  if  the  participant 
flies  directly  over  the  target  (within  88-90  degrees)  or  banks  the  aircraft  more  than  5  degrees  while 
the  camera  is  on  target. 

At  the  beginning  of  each  mission,  the  participant  is  initialized  at  the  southern  edge  of  the  tracker  map  at 
12,000  feet  heading  north  towards  the  target.  The  entire  area  is  cloud-covered,  except  for  a  single  break 
in  the  clouds  somewhere  nearby.  Several  resources  are  available  to  the  participant.  In  addition  to  the 
Heads-Up  Display  (Figure  9)  and  the  tracker  map  (Figure  10),  the  participant  has  a  choice  of  camera 
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Figure  9.  Heads-Up  Display 
overlaid  on  target  camera 
view.  In  this  illustration,  the 
target  (a  point  on  a  runway) 
is  almost  in  view.  While  in 
the  target  camera  view,  The 
reticle  (purple  cross-hair)  is 
always  pointed  at  the  target  on 
the  ground.  However,  as  illus¬ 
trated  in  this  example,  clouds 
often  obscure  the  target. 


views.  In  the  actual  Predator,  satellite  transmission  is  required  and  only  allows  enough  bandwidth 
for  one  video  feed.  Therefore  in  our  synthetic  task  the  participant  must  choose  to  use  either  the  EO 
camera  or  the  nose  camera.  The  participant  has  the  ability  to  switch  between  cameras  at  any  time; 
however,  the  EO  camera  must  be  in  use  to  obtain  TOT,  while  the  nose  camera  should  be  used  only 
to  aid  orientation. 

The  primary  measures  of  performance  for  each  trial  of  this  task  are:  total  time-on-target,  duration 
and  frequency  of  target-viewing  episodes,  and  duration  and  frequency  of  each  constraint  violation.  In 
addition,  after  completion  of  each  trial,  the  participant  is  asked  to  place  a  cursor  (the  red  arrow  in  Figure 
10)  where  he  or  she  believes  the  center  of  the  cloud  hole  to  be  located.  Distance  and  direction  and 
response  time  of  this  cursor  placement  from  the  veridical  cloud  hole  location  are  recorded.  The  feedback 
screen  shows  digital  values  for  the  total  time  on  target,  time  in  penalty,  hole  estimation  error  distance, 
number  of  bank  angle  and  depression  angle  infringements,  and  number  of  altitude,  airspeed,  and  no-fly 
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Figure  10.  The  tracker  map. 
The  target  is  depicted  in 
red  by  the  red  plus  symbol 
outlined  by  the  red  circle. 
The  large  yellow  box  rep¬ 
resents  a  no-fly  zone,  while 
the  large  blue  circle  outlines 
the  area  in  which  the  cloud 
hole  is  located.  Time  on 
target,  penalty  time,  time 
remaining,  altitude  con¬ 
straints,  slant  range,  depres¬ 
sion  angle,  and  bank  angle 
are  all  shown  to  the  partici¬ 
pant  in  the  gray  status  area. 


zone  violations.  The  synthetic  task  software  also  includes  a  program  that  displays  to  the  researcher  the 
path  flown  by  the  participant  superimposed  on  the  tracker  map  and  color-coded  to  reflect  key  events  such 
as  time  on  target  or  constraint  violations. 


Summary 

These  three  synthetic  tasks,  taken  together,  provide  a  way  to  measure  and  study  some  of  the  most 
critical  skills  required  to  pilot  a  Predator-like  UAV.  The  complete  software  necessary  to  run  these 
tasks,  including  data  logging  and  performance  measurement  is  available  to  Air  Force  Office  of 
Scientific  Research  (AFOSR)-supported  university  or  government  laboratories  upon  request.  For  more 
information,  or  to  request  the  software,  contact  Dr.  Elizabeth  Martin,  AFRL/HEA,  at  480-988-6561 

lelizabeth.martin@, williams. af.mill 
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APPENDIX  B:  SOFTWARE  DESCRIPTION 


Software  Architecture 


The  unmanned/uninhabited  aerial  vehicle  (UAV)  Simulation  Environment  (USE)  is  a  multiple-computer 
system  that  communicates  over  a  100  MHz  Ethernet  backbone.  The  USE  software  is  modular — it 
consists  of  nine  separate  programs,  written  in  C,  each  executing  individually.  Although  these  programs 
are  running  separately,  they  are  continually  communicating  with  one  another  through  sockets.  These 
sockets,  essentially  ports  within  each  program,  send  and  receive  data  over  the  network. 

The  main  program  coordinating  much  of  the  communication  is  called  use_os.  This  program  must  be 
running  on  each  PC  for  the  software  to  operate.  Use_os  is  the  administrator  for  the  USE  system  and 
coordinates  timing  and  keyboard  inputs. 


Each  program  represents  a  different  piece  of  functionality  for  the  system: 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 


USE_OS 

USE_HFUAV 

USE_SCSC 

USE_HFHCI 

USE_LFHCI 

USE_TM_HCI 

USE_PI 

USE_TRACKER 

USE_IOS 


Overall  administrator 

UAV  aerodynamics  /  vehicle  response 

Menu  system  /  ground  control  station  functionality 

High-fidelity  controls 

Low-fidelity  controls  (runs  on  researcher  computer) 
Low-fidelity  controls  (runs  on  participant  computer) 
3D  graphics  window 
Map  display 

Research  interface  /  instructor  operator  interface 


The  system  is  designed  to  reproduce  in  software  the  actual  flow  of  information  in  the  real  system.  The 
two  core  software  modules,  use_hfuav  and  use_scsc,  are  the  main  pieces  of  the  Predator  simulation. 

The  use_scsc  program  replicates  the  ground  control  station  (GCS)  from  the  actual  Predator  system.  The 
outward  interface  for  this  program  is  the  menu  system  used  in  the  GCS.  It  will  not  only  take  the  menu 
commands  and  act  upon  them,  it  accepts  data  from  other  processes,  aggregates  it  into  a  data  structure 
called  TIV,  and  transmits  to  the  process  representing  the  Predator  UAV.  The  use_hfuav  (hfuav  stands  for 
high-  fidelity  uav)  represents  the  actual  aircraft  in  the  system  and  receives  commands  and  information 
from  use_scsc.  The  software  reacts  to  the  commands  and  inputs,  and  reports  the  results,  in  a  structure 
called  VIT,  back  to  the  other  processes.  For  example,  if  the  user  makes  a  stick  input,  the  software  reading 
the  stick  sends  the  information  to  use_scsc  where  it  is  packaged  in  the  TIV.  This  information  is  then  sent 
to  use_hfuav  and  is  used,  along  with  other  state  information,  to  determine  the  aircraft’s  response  and  new 
state.  The  results  are  then  packaged  in  the  VIT  and  broadcast  over  the  net. 


Software  Listing 

Folders  and  functionality:  Each  number  is  a  folder  in  the  USE  directory.  The  files  in  each  directory 
follow  along  with  a  brief  description. 

1.  Configuration:  contains  various  configuration  files. 

Aero.txt:  allows  modification  of  aerodynamic  parameters.  Only  adjust  if  you  know 

what  you  are  doing.  Explanations  are  given. 

Default_dialog_color.txt:  changes  color  of  dialogs  presented  for  trainer.  Most  of  these  dialogs  appear 
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Fov.txt: 

Hci_deadband.txt: 
Hci_hybrid.txt: 
given  in  file. 

Left_emp.txt: 

Left_hfhci.txt: 

Left_omp.txt: 

Magvar.txt: 

Pi.txt: 

Powerup_config .  txt : 
Right_emp.txt: 

Right  _hfhci.txt: 

Right  _omp.txt: 
Uav_aero_coefficients.txt: 


when  using  the  pull-down  menus  on  the  Tracker  map. 
changes  field-of-view  and  other  sensor  settings, 
changes  stick  deadband. 

specifies  control  configuration  (i.e.,  research  or  high  fidelity).  Explanation 

specifies  emergency  mission  plan  waypoints  for  left  rack. 

file  containing  calibration  data  for  stick  (left  rack).  This  file  is  generated 

through  the  menu  system’s  calibration  area. 

specifies  operational  mission  plan  waypoints  for  left  rack. 

specifies  magnetic  variation. 

log  for  database  files  loaded  at  startup. 

log  for  scenario  file  loaded  at  startup. 

specifies  emergency  mission  plan  waypoints  for  right  rack. 

file  containing  calibration  data  for  stick  (right  rack). 

specifies  operational  mission  plan  waypoints  for  right  rack. 

specific  aircraft  characteristics  for  flying  (e.g.,  roll  limit).  Changing  these 

will  have  less  serious  affects  than  aero.txt  parameters.  These  are  more 

related  to  the  stability  augmentation  control  system  in  the  Predator  and  will 

be  used  mostly  to  limit  flight  performance. 


2.  Data:  contains  maps  for  the  Predator  ground  control  station.  You  should  store  your  maps  here 
for  consistency.  The  *.ga  is  a  format  used  by  the  contractor  who  manufacturers  the  Predator.  Other 
maps  generated  through  the  National  Imagery  and  Mapping  Agency’s  (NIMA)  tools  (e.g.  files  with  *.adr 
or  *.cab)  will  work.  You  will  need  arc  digitized  raster  graphic  (ADRG)  or  compressed  arc  digitized 
raster  graphic  (CADRG)  data,  and  will  use  the  NIMA  software  to  cut  out  the  desired  areas.  Check  out 
www.nima.mil  for  information  and  the  software  to  generate  your  own. 


3.  Databases:  contains  data  for  Distributed  Interactive  Simulation  (DIS),  databases  and  mission/ 
research  scenarios. 


Dis: 

Zhunter_liggett: 

Zindian_springs: 


Zscenarios: 


contains  DIS  configuration  files.  Not  necessary  for  research. 

Hunter  Liggett  database  (original).  Not  used. 

Indian  Springs  database  (original).  Used  for  Air  Force  Research  Labora¬ 
tory  (AFRL)  research  tasks.  This  area  contains  the  digital  terrain  elevation 
data  (DTED)  for  the  3D  database,  the  initial  ADRG  maps  loaded  at  startup, 
and  the  database’s  textures.  The  NIMA  software  generates  both  the  DTED 
and  the  ADRG  maps.  Using  the  software  and  data  CDs,  you  cut  out  the 
area  desired  and  place  it  in  this  file. 

scenario  files  for  tasks  and  training  scenarios.  Scenarios  are  for  AFRL 
research  tasks.  These  are  in  folders  labeled 

Basic_Maneuvering_Task_Leg_x,  Cloud_Break_Trial_Number_x,  and 
Landing_task  trial_numbcr_x.  The  (x)  denotes  trial  number.  The  files  in 
each  of  these  folders  is  self-explanatory  and  handles  some  portion  of  the 
initialization  for  the  task.  The  other  folders  contain  canned  scenarios  used 
in  the  trainer  (they  specify  starting  conditions) 
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4.  Executables:  contains  system  executables,  custom  *.dlls,  icons  and  *.cmd  (boot  commands) 
files  for  automation.  The  command  files  will  execute  the  programs  needed  for  the  function  specified 
in  their  title.  The  two  most  important  files  are  use_boot_scsc  and  use_boot_pso.  These  are  the  two 
necessary  to  start  the  research  software.  Use_boot_scsc.cmd  will  spawn  use_scsc.exe.  It  will  also  execute 
use_lfhci.exe,  and  use_ios.exe,  which  are  necessary  for  complete  functionality  on  the  menu  computer. 

It  may  also  spawn,  over  the  network,  use_boot_pso.cmd.  This  will  then  spawn  the  processes  necessary 
for  the  pilot  rack.  These  are  use_hfuav.exe,  use_ios.exe,  use_tm_hci.exe,  use_pi.exe,  and  use_tracker.exe. 
It  is  possible  to  edit  these  boot  command  files  to  add/delete  programs  started.  Simply  right  click  on  the 
files,  select  edit,  and  change  them  according  to  the  format  shown. 

5.  Help:  contains  help  files  that  appear  when  you  ask  for  help  through  the  trainer  software. 

6.  ITAR_and_Proprietary:  contains  copyright  and  International  Traffic  in  Arms  Regulation  (ITAR) 
messages  regarding  redistribution. 

7.  Logistics  Support:  contains  update  command  files  (*.cmd)  to  assist  in  updating  stations),  and 
runway  pictures  verifying  runway  alignment. 

8.  Missions:  contains  various  mission  plans  generated  through  the  mission  plan  functionality  of 
the  USE  software. 


9.  Presets:  contains  presets  files  for  the  software.  These  are  generated  through  the  Presets  area  of 
the  pulldown  menus. 


10.  Projects:  source  code  for  divisions  of  the  USE  software. 


Use_api: 


Use  dis: 


Use  hfhci: 


Use  ios: 


Use  lfhci: 


Use_pi: 


Use_pi_pi: 


Use  tm  hci: 


contains  the  header  and  library  files  for  the  USE  software.  You  will  make 
no  changes  in  this  file.  This  contains  the  main  data  structures  for  the 
software.  The  other  processes  expect  this  data  in  the  format.  Of  importance 
is  the  use_api.h  file.  This  file  contains  the  main  data  structures  used  in 
each  process.  Specifically,  it  contains  the  VIT  and  TIV  structures  that 
communicate  the  aircraft  state  information  across  the  net. 
contains  software  for  the  USE  implementation  of  the  DIS  protocol.  This 
is  not  fully  functional. 

software  to  read  inputs  from  the  high-fidelity  controls  on  the  actual  trainer. 

This  is  not  used  in  the  current  research  configuration. 

research  software.  This  is  where  all  the  tasks  are  programmed.  This  project 

spawns  the  process  that  is  the  interface  for  the  research. 

software  to  read  the  inputs  from  the  low-fidelity  controls  on  the  research 

station.  This  runs  on  the  instructor  operator  station  (IOS)  computer  and 

translates  the  low-fidelity  control  inputs  into  compatible  format  (the  hfhci 

format). 

3D  database  and  visuals  generation  software.  This  is  the  rest  of  the  render¬ 
ing  software.  Originally,  access  was  only  granted  through  the  plug-in. 
Using  these  two  modules,  one  has  access  to  all  the  rendering  source  code. 
3D  database  and  visuals  generation  software.  This  is  a  plug-in  for  the 
rendering  software  and  does  not  control  everything, 
second  software  module  to  read  the  input  from  the  low-fidelity  controls. 
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Use_tracker_pi: 


Use_tracker: 


This  must  run  on  the  pilot’s  /  subject’s  station.  It  will  take  the  inputs  and 
send  them  to  use_lfhci. 

tracker  map  software.  This  is  the  rest  of  the  map  software.  Originally, 
access  was  only  granted  through  the  plug-in.  Using  these  two  modules,  one 
has  access  to  all  the  map  source  code. 

tracker  map  software.  This  is  a  plug-in  for  tracker  map  software 
and  does  not  control  everything. 


Software  Modifiability 

The  types  of  modifications  that  can  be  accomplished  are  external  to  the  Predator-specific  simulation. 

The  two  core  software  packages,  use_hfuav  and  use_scsc,  are  inaccessible.  The  programmer  is  unable 
to  access  this  functionality  and  modify  it.  The  implications  of  this  are  that  researchers  are  unable  to 
modify  the  predator-specific  personality  of  the  aerodynamics  and  aircraft  response.  An  example  is  that  it 
is  impossible  to  modify  menu  commands  or  their  interactions  with  the  use_hfuav  process. 

The  basic  system  architecture  allows  a  programmer  to  tap  into  the  data  being  transmitted  on  the  different 
sockets  and  use  it  for  custom  applications.  This  translates  into  being  able  to  create  a  new  program  that 
traps  the  data  and  performs  some  action  desired  by  the  user.  For  example,  one  can  program  a  map  display 
and  position  the  aircraft  based  upon  the  data  received.  The  update  rate  of  this  data  is  20  ms  and  it  is 
controlled  by  a  heartbeat  from  the  use_os  application. 

With  the  exception  of  the  two  core  processes,  it  is  possible  to  replace  some  of  the  existing  applications 
with  new  ones.  An  application  is  transparent  to  the  others  as  long  as  it  transmits  any  data  the  others 
require.  For  example,  a  custom  3D  view  can  be  inserted  using  the  data  from  the  core  processes.  It  should 
be  mentioned  that  these  peripheral  processes  do  receive  data  from  the  core  applications  and  functionality 
could  be  implemented  to  act  upon  them.  Some  examples  are  the  reactions  the  3D  view  has  to  menu 
commands  from  the  SCSC  process.  A  less  invasive  possibility  is  to  modify  one  of  the  plug-in  or  parent 
modules.  The  plug-ins,  legacy  from  an  older  design,  allow  access  to  some  of  the  functionality,  like  the 
HUD.  The  parent  applications  use  the  plug-ins  and  their  own  code  to  generate  the  complete  program. 
This  methodology  is  used  in  the  map  and  3D  graphics  applications.  Using  these  it  is  possible  for  the 
programmer  to  modify  the  HUD,  Terrain  or  Map  display  for  a  customized  application.  Just  to  reiterate 
the  limitations,  the  effects  of  the  tracker  map  pull-down  menus  are  not  modifiable  because  they  rely 
upon  the  SCSC  module. 


Processes 
Hfuav 
3D  graphics 
Menu 

High-fidelity  controls 
Low-fidelity  controls  (menu) 
Low-fidelity  controls  (3D  graphics)  - 
Map 

Instructor  operator  station 


use_hfuav.exe 

use_pi.exe 


use_scsc.exe 

use_hfhci.exe 

use_lfhci.exe 


use_tm_hci.exe 


use_fiacker.exe 

use_ios.exe 
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Socket  Listing: 

7777  Left  Operator  Keyboard  Input:  transports  from  3D  graphics  computer  to  menu  computer 

7778  Right  Operator  Keyboard  Input:  transports  from  3D  graphics  computer  to  menu  computer 

8779  Left  Operator  Menu  Bar  commands:  transports  from  3D  graphics  process  to  menu  process 

8780  Left  Operator  menu  commands:  transports  from  menu  process  to  3D  graphics  process 

8781  Right  Operator  Menu  Bar  commands:  transports  from  3D  graphics  process  to  menu  process 

8782  Right  Operator  menu  commands:  transports  from  menu  process  to  3D  graphics  process 

8783  Left  Operator  map  presets:  transports  from  map  process  to  menu  process 

8784  Right  Operator  map  presets:  transports  from  map  process  to  menu  process 

8785  Left  Operator  GCS  data  (TIV):  transports  from  menu  process  to  map  process 

8786  Right  Operator  GCS  data  (TIV):  transports  from  menu  process  to  map  process 

8787  Left  Operator  vehicle  data  (VIT):  transports  from  hfuav  process  to  map  process 

8788  Right  Operator  vehicle  data  (VIT):  transports  from  hfuav  process  to  map  process 

8790  High/Low-fidelity  control  data  (analog  and  digital  inputs):  transports  from  high/low-fidelity 
controls  process  to  menu  process 

8791  High/Low-fidelity  control  data:  receive  port  for  digital  outputs 

8792  Left  Operator  vehicle  data  (VIT):  transports  from  hfuav  process  to  3D  graphics  process 

8793  Right  Operator  vehicle  data  (VIT):  transports  from  hfuav  process  to  3D  graphics  process 

8794  Left  Operator  GCS  data  (TIV):  transports  from  menu  process  to  3D  graphics  process 

8795  Right  Operator  GCS  data  (TIV):  transports  from  menu  process  to  3D  graphics  process 

8796  Left  Operator  GCS  data  (TIV):  transports  from  menu  process  to  instructor  operator  station 
process 

8797  Right  Operator  GCS  data  (TIV):  transports  from  menu  process  to  instructor  operator  station 
process 

8798  Left  Operator  vehicle  data  (VIT):  transports  from  hfuav  process  to  instructor  operator  station 
process 

8799  Right  Operator  vehicle  data  (VIT):  transports  from  hfuav  process  to  instructor  operator  station 
process 

8800  Left  operator  receive  port  for  digital  outputs:  transports  from  low-fidelity  controls  process  on  3D 
graphics  computer  to  low-fidelity  controls  process  on  menu  computer 

8801  Right  operator  receive  port  for  digital  outputs:  transports  from  low-fidelity  controls  process  on 
3D  graphics  computer  to  low-fidelity  controls  process  on  menu  computer 

8802  Left  operator  low  fidelity  control  data:  transports  from  low-fidelity  controls  process  on  3D 
graphics  computer  to  instructor  operator  station  process 

8803  Right  operator  low  fidelity  control  data:  transports  from  low-fidelity  controls  process  on  3D 
graphics  computer  to  instructor  operator  station  process 

8879  Left  Operator  vehicle  data  (VIT):  transports  from  hfuav  process  to  menu  process 

8880  Left  Operator  GCS  data  (TIV):  transports  from  menu  process  to  hfuav  process 

888 1  Right  Operator  vehicle  data  (VIT):  transports  from  hfuav  process  to  menu  process 

8882  Right  Operator  GCS  data  (TIV):  transports  from  menu  process  to  hfuav  process 

8884  Instructor  operator  station  commands:  transports  from  instructor  operator  station  process  to  3D 
graphics  process 

8885  Instructor  operator  station  commands:  transports  from  instructor  operator  station  process  to  map 
process 

8886  Instructor  operator  station  commands:  transports  from  instructor  operator  station  process  to  hfuav 
process 
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8887  Instructor  operator  station  commands:  transports  from  instructor  operator  station  process  to  high- 
fidelity  controls  process 

8888  Instructor  operator  station  commands:  transports  from  instructor  operator  station  process  to  menu 
process 

8892  Remote  Spawn  Commands 

8898  Raw  high-fidelity  control  data  (analog  and  digital  inputs):  transports  from  high-fidelity  controls 
process  to  instructor  operator  station  process 

8998  High/Low-fidelity  control  data  (analog  and  digital  inputs):  transmit  from  high/low-fidelity  con¬ 
trols  process  to  instructor  operator  station  process 


Data  Analysis  Software 

A  data  analysis  software  package  has  been  developed  to  facilitate  results  visualization  and  data  extrac¬ 
tion.  Written  in  C++,  this  software  uses  classes  to  ease  data  extraction.  A  class  has  been  developed  for 
each  task  (Basic  Maneuvering,  Landing,  and  Cloud  Break)  that  contains  the  functions  and  data  structures 
to  easily  access  trial  data.  All  source  code  is  available. 


Output  of  Summary  Data  Files 

The  synthetic  tasks  contain  built-in  performance  measurement  and  data  capture.  Each  trial 
produces  a  large  trial  data  file.  Selected  variables  from  the  trial  files  are  aggregated  into  a  summary  file, 
which  contains  one  line  of  data  per  subject  per  trial.  The  following  tables  show  the  variables  that  are 
included  in  these  summary  files  for  each  of  the  three  tasks. 


Basic  Maneuvering  Task  Summary  File 


cbp->research  subject  number_ Participant  number 


cbp->research  trial  number 

Participant  trial 

cbp->research  session  number 

Participant  session 

cbp->research  scenario  number 

Participant  scenario 

iFail  pass  trial  BM 

Pass  fail  flag  for  trial 

fAltitude  rms  error 

Altitude  RMS  error 

fHeadins  rms  error 

Heading  RMS  error 

fAirspeed  rms  error 

Airspeed  RMS  error 

roll  trim  success 

Number  of  successful  trims  in  roll 

roll  trim  failure 

Number  of  failed  trims  in  roll 

pitch  trim  success 

Number  of  successful  trims  in  pitch 

pitch  trim  failure 

Number  of  failed  trims  in  pitch 

fRe  search  bm  start  elev 

Start  altitude 

fRe  search  bm  start  head 

Start  heading 

fRe  search  bm  start  speed 

Start  speed 

fRe  search  bm  start  head 

Start  heading 

fResearch  bm  end  elev 

End  altitude 

fResearch  bm  end  head 

End  heading 

fResearch  bm  end  speed 

End  speed 

fResearch  bm  trial  duration  seconds 

Trial  duration  in  seconds 
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Landing  Task  Summary  File 


cbp->research  subiect  number 

Participant  number 

cbp->research  trial  number 

Participant  trial 

cbp->research  session  number 

Participant  session 

cbp->research  scenario  number 

Participant  scenario 

iPass  fail  flag 

Trial  Pass/Fail  flag 

dTouchdown  heading 

Heading  at  touchdown 

dTouchdown  groundspeed 

Groundspeed  at  touchdown 

dTouchdown  vsi 

VSI  at  touchdown 

dTouchdown  uav  x 

Uav  x  position  at  touchdown 

dTouchdown  uav  v 

Uav  v  position  at  touchdown 

dTouchdown  uav  z 

Uav  z  position  at  touchdown 

dTouchdown  uav  bank  an  ale 

Uav  bank  angle  at  touchdown 

dTouchdown  uav  lateral  movement 

Uav  lateral  movement  at  touchdown 

dTouchdown  pitch 

Uav  pitch  at  touchdown 

iAltitude_test[l] 

Results  of  altitude  test  at  gate  1  (0  =  fail, 

1  =  pass) 

iAltitude_test[2] 

Results  of  altitude  test  at  gate  2  (0  =  fail, 

1  =  pass) 

iAltitude_test[3] 

Results  of  altitude  test  at  gate  3  (0  =  fail, 

1  =  pass) 

dGroundtrack  approach  rms  error  LT 

Approach  groundtrack  RMS  error 

dGroundtrack  final  rms  error  LT 

Final  groundtrack  RMS  error 

dAltitude  rms  error  LT 

Altitude  RMS  error  on  final 

do  Total  trial  time  seconds 

Trial  time  in  seconds 

doApproach  time 

Time  in  Approach 

doFinal  time 

Time  in  Final 

roll  trim  success 

Number  of  successful  trims  in  roll 

roll  trim  failure 

Number  of  failed  trims  in  roll 

pitch  trim  success 

Number  of  successful  trims  in  pitch 

nitch  trim  failure 

Number  of  failed  trims  in  pitch 

iPass  heading  criteria 

Flag  to  indicate  Pass/Fail  of  heading  test 

iPass_groundspeed_criteria 

Flag  to  indicate  Pass/Fail  of  groundspeed 
test 

iPass  vsi  criteria 

Flag  to  indicate  Pass/Fail  of  vsi  test 

iPass_centerline_criteria 

Flag  to  indicate  Pass/Fail  of  centerline 
test 

iPass_uav_bank_angle_test 

Flag  to  indicate  Pass/Fail  of  bank  angle 
test 

iPass_lateral_velocity_test 

Flag  to  indicate  Pass/Fail  of  lateral 
velocity  test 

iPass  pitch  test 

Flag  to  indicate  Pass/Fail  of  pitch  test 

iPass  altitude  criteria 

Flag  to  indicate  Pass/Fail  of  altitude  test 

iPass  glideslope  criteria 

Flag  to  indicate  Pass/Fail  of  glideslope 

test 

iPass_groundtrack_criteria_approach 

Flag  to  indicate  Pass/Fail  of  groundtrack 
(approach)  test 
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iPas  s_groundtrack_criteria_final 

Flag  to  indicate  Pass/Fail  of  groundtrack 
(final)  test 

iUser_abort 

Flag  to  indicate  if  user  selected  to  abort 
test 

fRe  search  wind  speed 

Wind  speed 

fRe  search  wind  direction 

Wind  direction 

Reconnaissance  Task  Summary  File 

cbp->research_subject_number 

Participant  number 

cbp->research_trial_number 

Participant  trial 

cbp->research_session_number 

Participant  session 

cbp->research_scenario_number 

Participant  scenario 

etcbl_tx 

Subjects  cloud  hole  guess  x  coordinate 

etcbl_tz 

Subjects  cloud  hole  guess  z  coordinate 

tResearch_total_time_on_target 

TOT  with  minor  violations 

cbp->research_time_on_target 

TOT  with  no  violations  (no  minor  or  major) 

tResearch_cb_total_violation_time 

Total  violation  (major)  time 

dCloud_hole_click_distance 

Error  for  guess  of  cloud  hole  center 

roll_trim_success 

Number  of  successful  trims  in  roll 

roll_trim_failure 

Number  of  failed  trims  in  roll 

pitch_trim_success 

Number  of  successful  trims  in  pitch 

pitch_trim_failure 

Number  of  failed  trims  in  pitch 

cbp->research_uav_elev 

Start  altitude 

cbp->research_uav_head 

Start  heading 

cbp->research_uav_speed 

Start  speed 

cbp->research_hole_elev 

Hole  altitude 

cbp->research_hole_radius 

Hole  radius 

cbp->trial_duration_seconds 

Trial  duration  in  seconds 

tResearch_wind_speed 

Wind  speed 

tResearch_wind_direction 

Wind  direction 

cb_camera_first_hits_target_start_time 

When  timing  started  for  first  hit  counter  (should  be  ( 

cb_camera_first_hits_target_delta_time 

Time  from  start  of  trial  until  first  hit 

cb_on_target_time_count 

Total  number  of  on  target  events 

cbp->research_stall_violation_count 

Stall  violation  count 

cbp->research_altitude_violation_count 

Altitude  violation  count 

cbp->research_roz_violation_count 

ROZ  violation  count 

cb_bank_angle_violation_count 

Bank  angle  violation  count 

cb_operator_first_hits_violation_start_time 

When  timing  started  for  first  vio  counter  (should  be  1 

cb_operator_first_hits_violation_delta_time  Time  from  start  of  trial  until  first  violation 

cb_speed_flag_time_count 

Total  count  of  speed  violations 

cb_alt_flag_time_count 

Total  count  of  altitude  violations 

cb_roz_flag_time_c  ount 

Total  count  of  ROZ  violations 

cb_bank_angle_violation_count 

Total  count  of  bank  angle  violations 

cb_caged_time_count 

Total  count  of  caged  requests 

cb_pan_time_count 

Total  count  of  pan  requests 

cb_gimbal_time_count 

Total  count  of  gimbal  requests 
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APPENDIX  C:  DEMOGRAPHIC  QUESTIONNAIRE  AND  CONSENT  FORM 


INFORMED  CONSENT 

“Predator  UAV  Flying  Experience  Study” 

In  this  study  you  will  be  asked  to  participate  in  several  different  tasks  extended  over  multiple 
days.  Some  of  these  tasks  include  becoming  familiar  with  basic  flight  dynamics,  becoming  familiar  with 
Predator  Unmanned  Aerial  Vehicle  displays  and  controls,  flying  standard  maneuvers  in  a  simulator,  and 
flying  Predator-specific  tasks  in  a  simulator.  The  goal  of  this  study  is  to  assess  the  type  and  amount  of 
flying  training  necessary  to  operate  the  Predator  Unmanned  Aerial  Vehicle. 

There  are  no  risks,  physical  discomforts,  or  expected  emotional  discomforts  associated  with 
participation  in  this  study.  Your  participation  is  strictly  voluntary;  you  are  allowed  to  withdraw  from  the 
study  at  any  time  and  for  any  reason.  Furthermore,  your  performance  data  is  kept  strictly  confidential. 
The  data  will  be  stored  by  a  randomly  assigned  number  and  not  by  name.  The  data  will  then  be  pooled 
with  data  from  other  people  of  similar  experience.  We  are  interested  in  comparing  performance  amongst 
these  pooled  groups  of  data. 

Our  study  is  designed  to  evaluate  the  differences  in  performance  between  various  groups  of 
people.  If  other  potential  participants  are  infonned  of  the  purpose  of  the  study,  specifics  of  the  tasks,  or 
strategies  employed  during  the  study,  biases  may  emerge  in  our  data.  Therefore,  we  urge  you  to  avoid 
discussing  details  of  this  experiment  with  others.  Please  have  those  interested  direct  their  questions 
to  the  experimenter. 

Signature  on  this  consent  form  acknowledges  an  understanding  that  participation  is  voluntary  and 
is  on  a  non-interference  basis  with  Air  Force  duties. 

Finally,  we  wish  to  thank  you  for  your  participation  in  this  study! 


Name  (printed). 


Name  (signed). 


Date. 
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UAV  OPERATOR  STUDY  BACKGROUND  QUESTIONNAIRE 


Date _ Group_ _ Participant  # 

Age _ Sex  M  F 

Do  you  wear  corrective  lenses?  NO  GLASSES  CONTACTS 

Highest  education  level  attained: 

_ High  School 

_ Some  college,  but  no  degree 

_ Two-year  degree 

_ Four-year  degree  or  higher 


If  you  are  in  the  military: 

Grade:  _ 

Air  Force  Specialty  Code  _ 

Time  in  Present  Job  vrs _ mos 

Total  Active  Federal  Military  Duty  vrs _ mos 

Time  in  Career  Field  vrs _ mos 

Pilot  Experience 

What  military  pilot  training  have  you  had?  (please  check  all  that  apply) 

_ None 

_ Glider 

_ Ground  school 

_ T-3 

_ T-3  7 

_ T-3  8 

_ T-l 

_ .  Initial  qualification  in  an  operational  aircraft 

_ Mission  qualification  in  an  operational  aircraft 

_ Other  (please  specify _ _) 

What  private  pilot  training  have  you  had?  (please  check  all  that  apply) 
_ None 

_ Enrolled  in  ground  school  now 

_ Ground  school  for  private  pilot’s  certificate 

_ Private  pilot’s  certificate 

_ Instrument  rating 

_ Commercial  pilot  certificate 

_ Airline  transport  pilot  certificate 

_ Other  (please  specify _ ) 
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Please  list  the  number  of  hours  you  have  flown  particular  airplanes,  gliders,  other  airborne  craft,  or 
military  or  commercial  flight  simulators? 


#hrs 

#hrs 

#hrs 

#hrs 

#hrs 

#hrs 

#hrs 

#hrs 

#hrs 

#hrs 

#hrs 

#hrs 

What  is  the  most  recent  aircraft  you  have  flown,  and  when  was  the  last  time  you  flew  it? 


Date  last  flown _ 

What  is  the  most  recent  simulator  you  have  flown,  and  when  was  the  last  time  you  flew  it? 


Date  last  flown 


Miscellaneous  Interests 

On  average,  during  the  last  six  months,  how  many  hours  per  week  did  you  typically  spend  playing  flight 
simulation  video  games?. _ _  (enter  0  if  you  did  not  use  flight  simulation  games) 

Please  check  the  amount  of  lifetime  experience  you  have  playing  flight  simulation  games. 

_ None 

_ 1-10  hours 

_ 1 1-50  hours 

_ 50-200  hours 

_ Over  200  hours 

On  average,  during  the  last  six  months,  how  many  hours  per  week  did  you  typically  spend  playing 
3-D  action,  sports,  or  driving  video  games  (such  as  Doom,  Quake,  Tomb  Raider,  Madden  NFL,  FI 

Racing  Simulation — not  flight  simulation  games)?. _ _  (enter  0  if  you  did  not  play  3-D  action, 

sports,  or  driving  games) 
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Please  check  the  amount  of  lifetime  experience  you  have  playing  3-D  action/sports/driving  games. 

_ None 

_ 1-10  hours 

_ 1 1-50  hours 

_ 50-200  hours 

_ Over  200  hours 

On  average,  during  the  last  six  months,  how  many  hours  per  week  did  you  typically  spend  operating 

model  remote  control  vehicles  (model  airplanes,  boats,  cars,  etc.)? _ (enter  0  if  you  did  not 

operate  remote  control  vehicles) 

Please  check  the  amount  of  lifetime  experience  you  have  operating  remote  control  vehicles. 

_ None 

_ 1-10  hours 

_ 11-50  hours 

_ 50-200  hours 

_ Over  200  hours 

Please  feel  free  to  write  any  comments  you  may  have  about  the  study. 
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